Download Word - Terracotta Project Issue Tracker

[QTZ-385] Quartz is shutting down the jobstore before all jobs are completed
Created: 16/Jul/13 Updated: 28/Oct/16 Resolved: 25/Jul/13
Status:
Project:
Component/s:
Affects
Version/s:
Fix Version/s:
Closed
Quartz Scheduler (Historical - Do Not File New Issues Here - See GitHub)
Core
2.1.7
Type:
Reporter:
Resolution:
Labels:
Remaining
Estimate:
Time Spent:
Original
Estimate:
Bug
Steffen Ritter
Fixed
None
Not Specified
Issue Links:
Related
Terracotta
Target:
Fixed In
Revision:
BigMemory 4.0.4
2.2.1
Priority:
Assignee:
Votes:
2 Major
Chris Dennis
0
Not Specified
Not Specified
2370
Description
There is the possibility that the scheduler is shutting down the jobstore while a job is still running. This should n
down the scheduler with "sched.shutdown(true)", but it does. As a result, Quartz can't mark the job as "complete
WorkerThread will complete because of "shutdown(true)". If the job is marked for recovery it will be executed a
Error output is as follows:
ErrorLog
17:35:22,440 ERROR [org.quartz.core.ErrorLogger] - An error occured while marking executed
job= 'group_1.job_1'
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'myDS'
java.sql.SQLException: com.mchange.v2.c3p0.ComboPooledDataSource [ java.beans.Introspectio
java.lang.reflect.InvocationTargetException [lastAcquisitionFailureDefaultUser] ] has been
can no longer use it. [See nested exception: java.sql.SQLException:
com.mchange.v2.c3p0.ComboPooledDataSource [ java.beans.IntrospectionException:
java.lang.reflect.InvocationTargetException [lastAcquisitionFailureDefaultUser] ] has been
can no longer use it.]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.ja
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.jav
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.jav
at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSuppo
at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:
at org.quartz.core.JobRunShell.run(JobRunShell.java:285)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:567)
Caused by: java.sql.SQLException: com.mchange.v2.c3p0.ComboPooledDataSource [
java.beans.IntrospectionException: java.lang.reflect.InvocationTargetException
[lastAcquisitionFailureDefaultUser] ] has been closed() -- you can no longer use it.
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.assertCpds(AbstractPoolBackedDataSou
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getPoolManager(AbstractPoolBackedDat
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedData
at org.quartz.utils.PoolingConnectionProvider.getConnection(PoolingConnectionProvi
at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java
... 7 more
To overcome this problem I created a method for the SimpleThreadPool with some copied code from its own sh
SimpleThreadPool.java
public void waitForLastJobs() {
synchronized (nextRunnableLock) {
// Wait until all worker threads are shut down
while (busyWorkers.size() > 0) {
WorkerThread wt = (WorkerThread) busyWorkers.getFirst();
try {
getLog().debug(
"Waiting for thread " + wt.getName()
+ " to shut down");
// note: with waiting infinite time the
// application may appear to 'hang'.
nextRunnableLock.wait(2000);
} catch (InterruptedException ex) {
}
}
Iterator<WorkerThread> workerThreads = workers.iterator();
while (workerThreads.hasNext()) {
WorkerThread wt = (WorkerThread) workerThreads.next();
try {
wt.join();
workerThreads.remove();
} catch (InterruptedException ignore) {
}
}
getLog().debug("All 'LastJobWorker' were executed and stopped.");
}
}
Then I added a call of that method in the QuartzScheduler, just after joining the scheduler thread, but before the
down. My added code is between the two horizontal rulers.
QuartzScheduler.java
resources.getThreadPool().shutdown(waitForJobsToComplete);
if (waitForJobsToComplete) {
while (jobMgr.getNumJobsCurrentlyExecuting() > 0) {
try {
Thread.sleep(100);
} catch (Exception ignore) {
}
}
}
// Scheduler thread may have be waiting for the fire time of an acquired
// trigger and need time to release the trigger once halted, so make sure
// the thread is dead before continuing to shutdown the job store.
try {
schedThread.join();
} catch (InterruptedException ignore) {
}
//-----------------if(waitForJobsToComplete) {
((SimpleThreadPool) resources.getThreadPool()).waitForLastJobs();
}
//-----------------closed = true;
if (resources.getJMXExport()) {
try {
unregisterJMX();
} catch (Exception e) {
}
}
if(boundRemotely) {
try {
unBind();
} catch (RemoteException re) {
}
}
shutdownPlugins();
resources.getJobStore().shutdown();
I'm no expert, maybe there is a better way to do this but for now it works for me and the errors are gone. Quartz
until all jobs are done. Maybe my input is useful for someone who can look into it and use it somehow for the ne
Comments
Comment by Chris Dennis [ 25/Jul/13 ]
I've reworked the scheduler shutdown procedure to make sure the scheduler thread is shutdown
before the pool. This way when shutting down the pool we are guaranteed there can be no late
arriving jobs that miss the pool shutdown.
Comment by Yakov Feldman [ 29/Jul/13 ]
Hi Chris,
I see only trunk change 2370. Did you merge this change to 4.0.x and 4.1.x?
Comment by Yakov Feldman [ 30/Jul/13 ]
Hi Chris,
Can you please provide steps to reproduce this defect.
Comment by Chris Dennis [ 31/Jul/13 ]
Quartz doesn't have dedicated 4.1/4.0 branches, trunk is the destined for 4.0. There is nowhere
to merge these changes to.
Comment by Chris Dennis [ 31/Jul/13 ]
Reproducing this is not easy. I could not reproduce the users reported failure mode, but by code
inspection I could understand how it could happen, and what the correct resolution was to
prevent this in the future. I have the task of writing a unit test to cover this on my todo list, but
this will not be an easy task (plenty of things to mock and coordinate).
Comment by Chris Dennis [ 01/Aug/13 ]
I have added QTZ385Test that covers this behavior.
Comment by Satish Satish [ 08/Aug/13 ]
Hi Chris,
Since this is difficult to reproduce what do you think is the best way for us to verify this bug?
Comment by Yakov Feldman [ 18/Aug/13 ]
Verified with func_QuartzTest_4.0:ShutdownWithWaitTest
Comment by Sascha Scholz [ 28/Oct/16 ]
Did this maybe cause the regression that was reported in QTZ-493 and QTZ-504?
Generated at Sun May 14 16:19:30 PDT 2017 using JIRA 6.2.4#6261sha1:4d2e6f6f26064845673c8e7ffe9b6b84b45a6e79.