Download Word - Camunda

[CAM-3303] Job/incident exception message is not populated when exception is
from a transaction synchronization Created: 13/Jan/15 Updated: 26/Oct/16 Resolved: 19/Oct/16
Status:
Project:
Component/s:
Affects
Version/s:
Fix Version/s:
Closed
camunda BPM
engine
None
Type:
Reporter:
Resolution:
Labels:
Remaining
Estimate:
Time Spent:
Original
Estimate:
Bug Report
Thorben Lindhauer
Fixed
SUPPORT
0 minutes
Issue Links:
Depedendency
Related
is related to CAM-6751 In Glassfish, a TransactionListener i...
is related to CAM-6727 Fix spring unit test suite memory con...
7.6.0, 7.3.8, 7.6.0-alpha4, 7.4.10, 7.5.6
Priority:
Assignee:
Votes:
L3 - Default
Daniel Meyer
0
Not Specified
0 minutes
Open
Closed
Description
Problem:
When a job is executed within a JTA transaction and an exception is thrown upon commit (i.e.
invocation of JtaTransactionInterceptor#doCommit), the job's retries are decremented properly
but the job's exception message (and incident message) is not populated.
Reason:
Job retries are implemented via a transaction listener. This listener is handed the exception
when it is thrown. However, this is only done by the CommandContextInterceptor, meaning
that only exceptions thrown from within a command or on its flush are treated that way.
Exceptions thrown from a transacrtion synchronization (transaction listener) cannot be caught
this way. A common example is JPA which often implements manged flush of the transaction
scoped entity manager as TX-Sync on commit. Exceptions thrown by JPA at this point are not
visible in the process engine database. Those kinds of exceptions are then only logged in
ExecuteJobsRunnable#run.
Possible solution:
Implement job retry decrements in ExecuteJobsRunnable#run instead of the transaction listener.
Then, the catched exception can not only be logged but also set on the job. However, we have
to make sure that we do not change the existing behavior other than that. Especially an
exception thrown within the job command should not be wrapped in any other exception by a
command interceptor. Thus, there has to be some way of handing these "out", for example by
setting them on the current command.
Comments
Comment by Sebastian Menski [ 11/Aug/16 ]
Hi Philipp Ossler,
the integrations fail, see for example:
https://hq2.camunda.com/jenkins/ci/view/Broken/job/7.6-engine-IT-wildfly-82-h2/218/testReport/
https://hq2.camunda.com/jenkins/ci/view/Broken/job/7.6-EE-engine-IT-jbosseap-70-h2/221/testReport/
https://hq2.camunda.com/jenkins/ci/view/Broken/job/7.6-EE-engine-IT-weblogic-12R1-h2/279/testReport/
Comment by Sebastian Menski [ 12/Aug/16 ]
It seems that the integration tests fail on all jboss, wildfly, websphere and weblogic builds.
Comment by Sebastian Menski [ 13/Aug/16 ]
I revert all changes for this ticket:
https://github.com/camunda/camunda-bpm-platform/commit/da63f241c7b57fe980249ac5ae3be369648d3782
Comment by Sebastian Menski [ 07/Sep/16 ]
It seems that the ConcurrentJobExecutorTest does change something related to the job execution and does not cl
RuntimeServiceAsyncOperationsTest which than hangs.
"main" prio=10 tid=0x00007f1270006000 nid=0x1c4b in Object.wait() [0x00007f127670f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000eae3c030> (a org.camunda.bpm.engine.test.concurrency.Concur
at java.lang.Object.wait(Object.java:485)
at org.camunda.bpm.engine.test.concurrency.ControllableThread.returnControlToTestTh
- locked <0x00000000eae3c030> (a org.camunda.bpm.engine.test.concurrency.Concurrent
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecu
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(Comman
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.exe
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.ja
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.test.concurrency.ConcurrentJobExecutorTest$ControllableEx
at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.executeJob(ExecuteJobsCmd.java:12
at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:93)
at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecu
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(Comman
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.exe
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.ja
at org.camunda.bpm.engine.impl.ManagementServiceImpl.executeJob(ManagementServiceIm
at
org.camunda.bpm.engine.test.api.runtime.RuntimeServiceAsyncOperationsTest.testDeleteProces
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:4
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedEx
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.camunda.bpm.engine.test.ProcessEngineRule$1.evaluate(ProcessEngineRule.java:
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.ja
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(Fo
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.ja
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Locked ownable synchronizers:
- None
Comment by Sebastian Menski [ 07/Sep/16 ]
Hi Philipp Ossler,
does this fix requires more connections to the database? As we see issues in the engine-it suite on oracle and sqls




https://hq2.camunda.com/jenkins/ci/job/7.6-engine-IT-XA-wildfly-100-sqlserver2014/44/testReport/org.camunda.bpm.integrationtest.functional.classloading.variables/SerializableVariab
https://hq2.camunda.com/jenkins/ci/job/7.6-EE-engine-IT-jbosseap-62-oracle10/lastCompletedBuild/testReport/junit/org.camunda.bpm.integrationtest.functional.ejb.remote/RemoteS
https://hq2.camunda.com/jenkins/ci/job/7.6-EE-engine-IT-jbosseap-63-oracle10/44/testReport/junit/org.camunda.bpm.integrationtest.functional.ejb.remote/RemoteSFSBInvocationTe
https://hq2.camunda.com/jenkins/ci/job/7.6-engine-IT-XA-jboss-72-oracle10/43/testReport/junit/org.camunda.bpm.integrationtest.functional.ejb.remote/RemoteSFSBInvocationTe
/cc Christian Lipphardt
Comment by Sebastian Menski [ 07/Sep/16 ]
Also we got an out of memory issue. Could this be related to this issue?

https://hq2.camunda.com/jenkins/ci/view/Broken/job/7.6-platform-jdks/95/container=h2,jdk=ibm-jdk-6latest/testReport/junit/org.camunda.bpm.engine.spring.test.transaction/SpringTransactionIntegrationTest/
Comment by Sebastian Menski [ 08/Sep/16 ]
I created a job which will hopefully generate a heap dump on out of memory if I configured everything correctly
https://hq2.camunda.com/jenkins/ci/job/TEST-ibm-jdk-heapdump/
Comment by Sebastian Menski [ 08/Sep/16 ]
Hi Philipp Ossler, Daniel Meyer, Christian Lipphardt,
I debugged at least the memory issue and found out that the engine-spring test suites creates a lot of engines with
whatever reason also leads to a lot more engines. I found 9 engines at the end of the test. I think this is a problem
deleted.
I create for that issue. Until this the jdk ibm test suite will fail on odd hours.
@Daniel Meyer could you decided on the importance of ?
Cheers,
Sebastian
PS: We think the connection issue is also somehow related to this but I didn't had time to debug it. Also we may
Comment by Daniel Meyer [ 11/Oct/16 ]
Just found a problem here: on the following servers, the MDB* is not invoked anymore: WLS, WAS, GF

org.camunda.bpm.container.impl.ejb.JobExecutionHandlerMDB
Generated at Thu Jun 08 03:19:38 CEST 2017 using JIRA 6.4.6#64021sha1:33e5b454af4594f54560ac233c30a6e00459507e.