Download [#MULE-4387] Intermittent test failure

[MULE-4387] Intermittent test failure: ServiceInFlightMessagesTestCase Created:
05/Jun/09 Updated: 05/Apr/16 Resolved: 05/Apr/16
Status:
Project:
Component/s:
Affects
Version/s:
Fix Version/s:
Closed
Mule
Build: Integration Tests, Core: Queues (SEDA) / Persistence
2.2.3 (EE only)
Type:
Reporter:
Resolution:
Labels:
Remaining
Estimate:
Time Spent:
Original
Estimate:
Bug
Daniel Zapata
Won't Fix or Usage Issue
None
Not Specified
None
Priority:
Assignee:
Votes:
Major
Daniel Feist
0
Not Specified
Not Specified
Issue
Duplicate
Links:
Medium
User
impact:
==========================================================================
Log
Output: = Testing: in flight messages =
= (org.mule.test.integration.service.ServiceInFlightMessagesTestCase) =
=========================================================================
[06-05 10:21:38] ERROR DefaultExceptionStrategy [TestService.15]: Caught exception in Exception S
java.io.IOException: Failed to create directory: /home/dzapata/code/ce-mule3.x/tests/integration/.mule/queuestore/out/df76c12c-0e91-4923-8fdd-b334ab8effc7.msg
java.lang.RuntimeException: java.io.IOException: Failed to create directory: /home/dzapata/code/ce-m
3.x/tests/integration/.mule/queuestore/out/df76c12c-0e91-4923-8fdd-b334ab8effc7.msg
at org.mule.util.queue.TransactionalQueueSession$QueueImpl.offer(TransactionalQueueSession.java:9
at org.mule.util.queue.TransactionalQueueSession$QueueImpl.put(TransactionalQueueSession.java:56
at org.mule.transport.vm.VMMessageDispatcher.doDispatch(VMMessageDispatcher.java:56)
at org.mule.transport.vm.VMMessageDispatcher.doSend(VMMessageDispatcher.java:104)
at org.mule.transport.AbstractMessageDispatcher.send(AbstractMessageDispatcher.java:163)
at org.mule.transport.AbstractConnector.send(AbstractConnector.java:2088)
at org.mule.endpoint.DefaultOutboundEndpoint.send(DefaultOutboundEndpoint.java:76)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleSession.java:327)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleSession.java:213)
at org.mule.routing.outbound.AbstractOutboundRouter$2.doInTransaction(AbstractOutboundRouter.ja
at org.mule.transaction.TransactionTemplate.execute(TransactionTemplate.java:99)
at org.mule.routing.outbound.AbstractOutboundRouter.send(AbstractOutboundRouter.java:159)
at org.mule.routing.outbound.FilteringOutboundRouter.route(FilteringOutboundRouter.java:79)
at org.mule.routing.outbound.OutboundPassThroughRouter.route(OutboundPassThroughRouter.java:6
at
org.mule.routing.outbound.DefaultOutboundRouterCollection$1.doInTransaction(DefaultOutboundRo
at org.mule.transaction.TransactionTemplate.execute(TransactionTemplate.java:45)
at org.mule.routing.outbound.DefaultOutboundRouterCollection.route(DefaultOutboundRouterCollecti
at org.mule.service.AbstractService.dispatchToOutboundRouter(AbstractService.java:868)
at org.mule.model.seda.SedaService.dispatchToOutboundRouter(SedaService.java:558)
at org.mule.model.seda.SedaService$ComponentStageWorker.run(SedaService.java:581)
at org.mule.work.WorkerContext.run(WorkerContext.java:310)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Failed to create directory: /home/dzapata/code/ce-mule3.x/tests/integration/.mule/queuestore/out/df76c12c-0e91-4923-8fdd-b334ab8effc7.msg
at org.mule.util.queue.FilePersistenceStrategy.store(FilePersistenceStrategy.java:66)
at org.mule.util.queue.TransactionalQueueManager.doStore(TransactionalQueueManager.java:237)
at org.mule.util.queue.TransactionalQueueSession$QueueImpl.offer(TransactionalQueueSession.java:7
... 23 more
[06-05 10:21:38] ERROR DefaultServiceExceptionStrategy [TestService.15]:
********************************************************************************
Message : Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=vm://out,
connector=VMConnector{this=449afc, started=true, initialised=true, name='vmOutPersistent', disposed
numberOfConcurrentTransactedReceivers=4, createMultipleTransactedReceivers=true, connected=true
supportedProtocols=[vm], serviceOverrides=null}, transformer=[], name='out', properties={},
transactionConfig=Transaction{factory=null, action=NEVER, timeout=0}, filter=null, deleteUnaccepte
securityFilter=null, synchronous=true, initialState=started, responseTimeout=10000, endpointEncoding
payload is of type: String
Type : org.mule.api.transport.DispatchException
Code : MULE_ERROR-42999
JavaDoc : http://www.mulesource.org/docs/site/current2/apidocs/org/mule/api/transport/DispatchExcep
Payload : test
********************************************************************************
Exception stack is:
1. Failed to create directory: /home/dzapata/code/ce-mule-3.x/tests/integration/.mule/queuestore/out/df7
8fdd-b334ab8effc7.msg (java.io.IOException)
org.mule.util.queue.FilePersistenceStrategy:66 (http://java.sun.com/j2se/1.5.0/docs/api/java/io/IOExce
2. java.io.IOException: Failed to create directory: /home/dzapata/code/ce-mule3.x/tests/integration/.mule/queuestore/out/df76c12c-0e91-4923-8fdd-b334ab8effc7.msg (java.lang.Run
org.mule.util.queue.TransactionalQueueSession$QueueImpl:91
(http://java.sun.com/j2se/1.5.0/docs/api/java/lang/RuntimeException.html)
3. Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=vm://out,
connector=VMConnector{this=449afc, started=true, initialised=true, name='vmOutPersistent', disposed
numberOfConcurrentTransactedReceivers=4, createMultipleTransactedReceivers=true, connected=true
supportedProtocols=[vm], serviceOverrides=null}, transformer=[], name='out', properties={},
transactionConfig=Transaction{factory=null, action=NEVER, timeout=0}, filter=null, deleteUnaccepte
securityFilter=null, synchronous=true, initialState=started, responseTimeout=10000, endpointEncoding
payload is of type: String (org.mule.api.transport.DispatchException)
org.mule.transport.AbstractMessageDispatcher:185
(http://www.mulesource.org/docs/site/current2/apidocs/org/mule/api/transport/DispatchException.html
********************************************************************************
Root Exception stack trace:
java.io.IOException: Failed to create directory: /home/dzapata/code/ce-mule3.x/tests/integration/.mule/queuestore/out/df76c12c-0e91-4923-8fdd-b334ab8effc7.msg
at org.mule.util.queue.FilePersistenceStrategy.store(FilePersistenceStrategy.java:66)
at org.mule.util.queue.TransactionalQueueManager.doStore(TransactionalQueueManager.java:237)
at org.mule.util.queue.TransactionalQueueSession$QueueImpl.offer(TransactionalQueueSession.java:7
at org.mule.util.queue.TransactionalQueueSession$QueueImpl.put(TransactionalQueueSession.java:56
at org.mule.transport.vm.VMMessageDispatcher.doDispatch(VMMessageDispatcher.java:56)
at org.mule.transport.vm.VMMessageDispatcher.doSend(VMMessageDispatcher.java:104)
at org.mule.transport.AbstractMessageDispatcher.send(AbstractMessageDispatcher.java:163)
at org.mule.transport.AbstractConnector.send(AbstractConnector.java:2088)
at org.mule.endpoint.DefaultOutboundEndpoint.send(DefaultOutboundEndpoint.java:76)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleSession.java:327)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleSession.java:213)
at org.mule.routing.outbound.AbstractOutboundRouter$2.doInTransaction(AbstractOutboundRouter.ja
at org.mule.transaction.TransactionTemplate.execute(TransactionTemplate.java:99)
at org.mule.routing.outbound.AbstractOutboundRouter.send(AbstractOutboundRouter.java:159)
at org.mule.routing.outbound.FilteringOutboundRouter.route(FilteringOutboundRouter.java:79)
at org.mule.routing.outbound.OutboundPassThroughRouter.route(OutboundPassThroughRouter.java:6
at
org.mule.routing.outbound.DefaultOutboundRouterCollection$1.doInTransaction(DefaultOutboundRo
at org.mule.transaction.TransactionTemplate.execute(TransactionTemplate.java:45)
at org.mule.routing.outbound.DefaultOutboundRouterCollection.route(DefaultOutboundRouterCollecti
at org.mule.service.AbstractService.dispatchToOutboundRouter(AbstractService.java:868)
at org.mule.model.seda.SedaService.dispatchToOutboundRouter(SedaService.java:558)
at org.mule.model.seda.SedaService$ComponentStageWorker.run(SedaService.java:581)
at org.mule.work.WorkerContext.run(WorkerContext.java:310)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
at java.lang.Thread.run(Thread.java:595)
********************************************************************************
[06-05 10:21:38] ERROR DefaultServiceExceptionStrategy [TestService.15]:
********************************************************************************
Message : Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=vm://out,
connector=VMConnector{this=449afc, started=true, initialised=true, name='vmOutPersistent', disposed
numberOfConcurrentTransactedReceivers=4, createMultipleTransactedReceivers=true, connected=true
supportedProtocols=[vm], serviceOverrides=null}, transformer=[], name='out', properties={},
transactionConfig=Transaction{factory=null, action=NEVER, timeout=0}, filter=null, deleteUnaccepte
securityFilter=null, synchronous=true, initialState=started, responseTimeout=10000, endpointEncoding
payload is of type: String
Type : org.mule.api.transport.DispatchException
Code : MULE_ERROR-42999
JavaDoc : http://www.mulesource.org/docs/site/current2/apidocs/org/mule/api/transport/DispatchExcep
Payload : test
********************************************************************************
Exception stack is:
1. Failed to create directory: /home/dzapata/code/ce-mule-3.x/tests/integration/.mule/queuestore/out/df7
8fdd-b334ab8effc7.msg (java.io.IOException)
org.mule.util.queue.FilePersistenceStrategy:66 (http://java.sun.com/j2se/1.5.0/docs/api/java/io/IOExce
2. java.io.IOException: Failed to create directory: /home/dzapata/code/ce-mule3.x/tests/integration/.mule/queuestore/out/df76c12c-0e91-4923-8fdd-b334ab8effc7.msg (java.lang.Run
org.mule.util.queue.TransactionalQueueSession$QueueImpl:91
(http://java.sun.com/j2se/1.5.0/docs/api/java/lang/RuntimeException.html)
3. Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=vm://out,
connector=VMConnector{this=449afc, started=true, initialised=true, name='vmOutPersistent', disposed
numberOfConcurrentTransactedReceivers=4, createMultipleTransactedReceivers=true, connected=true
supportedProtocols=[vm], serviceOverrides=null}, transformer=[], name='out', properties={},
transactionConfig=Transaction{factory=null, action=NEVER, timeout=0}, filter=null, deleteUnaccepte
securityFilter=null, synchronous=true, initialState=started, responseTimeout=10000, endpointEncoding
payload is of type: String (org.mule.api.transport.DispatchException)
org.mule.transport.AbstractMessageDispatcher:185
(http://www.mulesource.org/docs/site/current2/apidocs/org/mule/api/transport/DispatchException.html
********************************************************************************
Root Exception stack trace:
java.io.IOException: Failed to create directory: /home/dzapata/code/ce-mule3.x/tests/integration/.mule/queuestore/out/df76c12c-0e91-4923-8fdd-b334ab8effc7.msg
at org.mule.util.queue.FilePersistenceStrategy.store(FilePersistenceStrategy.java:66)
at org.mule.util.queue.TransactionalQueueManager.doStore(TransactionalQueueManager.java:237)
at org.mule.util.queue.TransactionalQueueSession$QueueImpl.offer(TransactionalQueueSession.java:7
at org.mule.util.queue.TransactionalQueueSession$QueueImpl.put(TransactionalQueueSession.java:56
at org.mule.transport.vm.VMMessageDispatcher.doDispatch(VMMessageDispatcher.java:56)
at org.mule.transport.vm.VMMessageDispatcher.doSend(VMMessageDispatcher.java:104)
at org.mule.transport.AbstractMessageDispatcher.send(AbstractMessageDispatcher.java:163)
at org.mule.transport.AbstractConnector.send(AbstractConnector.java:2088)
at org.mule.endpoint.DefaultOutboundEndpoint.send(DefaultOutboundEndpoint.java:76)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleSession.java:327)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleSession.java:213)
at org.mule.routing.outbound.AbstractOutboundRouter$2.doInTransaction(AbstractOutboundRouter.ja
at org.mule.transaction.TransactionTemplate.execute(TransactionTemplate.java:99)
at org.mule.routing.outbound.AbstractOutboundRouter.send(AbstractOutboundRouter.java:159)
at org.mule.routing.outbound.FilteringOutboundRouter.route(FilteringOutboundRouter.java:79)
at org.mule.routing.outbound.OutboundPassThroughRouter.route(OutboundPassThroughRouter.java:6
at
org.mule.routing.outbound.DefaultOutboundRouterCollection$1.doInTransaction(DefaultOutboundRo
at org.mule.transaction.TransactionTemplate.execute(TransactionTemplate.java:45)
at org.mule.routing.outbound.DefaultOutboundRouterCollection.route(DefaultOutboundRouterCollecti
at org.mule.service.AbstractService.dispatchToOutboundRouter(AbstractService.java:868)
at org.mule.model.seda.SedaService.dispatchToOutboundRouter(SedaService.java:558)
at org.mule.model.seda.SedaService$ComponentStageWorker.run(SedaService.java:581)
at org.mule.work.WorkerContext.run(WorkerContext.java:310)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
at java.lang.Thread.run(Thread.java:595)
Similar
Issues:
Description
The CE bamboo build has been failing intermittently on ServiceInFlightMessagesTestCase:
http://bamboo.mulesource.org/build/viewBuildResultsFailedTests.action?buildKey=MULE-MULE3JDK5&buil
http://bamboo.mulesource.org/build/viewBuildResultsFailedTests.action?buildKey=MULE-MULE3JDK5&buil
http://bamboo.mulesource.org/build/viewBuildResultsFailedTests.action?buildKey=MULE-MULE3JDK5&buil
http://bamboo.mulesource.org/build/viewBuildResultsFailedTests.action?buildKey=MULE-MULE3JDK5&buil
http://bamboo.mulesource.org/build/viewBuildResultsFailedTests.action?buildKey=MULE-MULE3JDK5&buil
http://bamboo.mulesource.org/build/viewBuildResultsFailedTests.action?buildKey=MULE-MULE3JDK5&buil
http://bamboo.mulesource.org/build/viewBuildResultsFailedTests.action?buildKey=MULE-MULE3JDK5&buil
In flight stop persistent messages paused service
expected:<500> but was:<499>
junit.framework.AssertionFailedError: expected:<500> but was:<499>
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:280)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:198)
at junit.framework.Assert.assertEquals(Assert.java:204)
at org.mule.test.integration.service.ServiceInFlightMessagesTestCase.assertNoLostMessages(ServiceInFlightMe
at
org.mule.test.integration.service.ServiceInFlightMessagesTestCase.testInFlightStopPersistentMessagesPausedSe
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:164)
at junit.framework.TestCase.runBare(TestCase.java:130)
at org.mule.tck.AbstractMuleTestCase.runBare(AbstractMuleTestCase.java:255)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:120)
at org.mule.tck.AbstractMuleTestCase.run(AbstractMuleTestCase.java:235)
at junit.framework.TestSuite.runTest(TestSuite.java:230)
at junit.framework.TestSuite.run(TestSuite.java:225)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.maven.surefire.junit.JUnitTestSet.execute(JUnitTestSet.java:213)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:1
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165)
at org.apache.maven.surefire.Surefire.run(Surefire.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:993)
1 sec
In flight messages
expected:<500> but was:<499>
junit.framework.AssertionFailedError: expected:<500> but was:<499>
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:280)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:198)
at junit.framework.Assert.assertEquals(Assert.java:204)
at org.mule.test.integration.service.ServiceInFlightMessagesTestCase.assertNoLostMessages(ServiceInFlightMe
at org.mule.test.integration.service.ServiceInFlightMessagesTestCase.testInFlightMessages(ServiceInFlightMes
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:164)
at junit.framework.TestCase.runBare(TestCase.java:130)
at org.mule.tck.AbstractMuleTestCase.runBare(AbstractMuleTestCase.java:255)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:120)
at org.mule.tck.AbstractMuleTestCase.run(AbstractMuleTestCase.java:235)
at junit.framework.TestSuite.runTest(TestSuite.java:230)
at junit.framework.TestSuite.run(TestSuite.java:225)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.maven.surefire.junit.JUnitTestSet.execute(JUnitTestSet.java:213)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:1
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165)
at org.apache.maven.surefire.Surefire.run(Surefire.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:993)
I tried adding an assert to make sure that mule was stopped before the message count was checked in assertNoLo
assertFalse("Mule needs to be stopped before we can check the message count", muleContext.isStarted());
However, I saw the failure again without triggering the exception.
See the log output section for an additional exception I noticed when I reproduced this error locally. This may gi
Comments
Comment by Daniel Zapata [ 05/Jun/09 ]
This is probably related to our graceful shutdown story and is causing intermittent build failures.
Comment by Daniel Zapata [ 05/Oct/09 ]
This is failing on the 2.2.x build:
http://bamboo.mulesoft.org/browse/MULE-MULE22JDK5-364
Comment by Andrew Perepelytsya [ 26/Feb/10 ]
I and DF spent some time on it, our findings showed that the race condition is at the
muleContext startup, when 2 competing TransactionalQueueManagers (together with their
FilePersistentStrategies) are introduced. This is just a mistake in the test, and we'll look into
correcting the test.
Other than that, it's all about timing (we're trying to interrupt Mule when it moves messages
from seda queue -> vm queue), also can be improved.
P.S.: DF assigning it to you simply because can't have 2 assignees
Comment by Matias Baldini [ 05/Apr/16 ]
Mule issues cleanup.
Generated at Wed Jun 07 21:20:01 CDT 2017 using JIRA 7.2.4#72006sha1:91606a849e49df24789f9d9a6614de45c98c37a8.