Can I make waiting Quartz Jobs fire in the order they were triggered?

Question

I have an application using the Quartz Scheduler to schedule jobs. The application is currently running Quartz version 1.6.2. My JobStore is org.quartz.impl.jdbcjobstore.JobStoreTX with an Oracle database backing it. Clustering is turned on, but there's only one scheduler using the database. My Quartz threadPool is configured as follows:

org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 5
org.quartz.threadPool.threadPriority = 5

My jobs are long running, so it's fairly common to have 5 jobs running (the maximum allowed by my thead pool) when triggers fire new jobs. The newly triggered jobs misfire and I see log messages like the following:

2011-05-20 04:09:30,097 INFO  [QuartzScheduler_scheduler-servername-111305822374881_MisfireHandler] o.q.p.h.LoggingTriggerHistoryPlugin - Trigger DEFAULT.JobName1 misfired job DEFAULT.DEFAULT  at:  04:09:30 05/20/2011.  Should have fired at:  04:08:29 05/20/2011
2011-05-20 04:09:30,120 INFO  [QuartzScheduler_scheduler-servername-111305822374881_MisfireHandler] o.q.p.h.LoggingTriggerHistoryPlugin - Trigger DEFAULT.JobName1 misfired job DEFAULT.DEFAULT  at:  04:09:30 05/20/2011.  Should have fired at:  04:09:30 05/20/2011
2011-05-20 04:09:30,125 INFO  [QuartzScheduler_scheduler-servername-111305822374881_MisfireHandler] o.q.p.h.LoggingTriggerHistoryPlugin - Trigger DEFAULT.JobName2 misfired job DEFAULT.DEFAULT  at:  04:09:30 05/20/2011.  Should have fired at:  04:08:30 05/20/2011
2011-05-20 04:09:30,138 INFO  [QuartzScheduler_scheduler-servername-111305822374881_MisfireHandler] o.q.p.h.LoggingTriggerHistoryPlugin - Trigger DEFAULT.JobName2 misfired job DEFAULT.DEFAULT  at:  04:09:30 05/20/2011.  Should have fired at:  04:09:30 05/20/2011
2011-05-20 04:11:29,998 INFO  [QuartzScheduler_scheduler-servername-111305822376676_MisfireHandler] o.q.impl.jdbcjobstore.JobStoreTX - Handling 2 trigger(s) that missed their scheduled fire-time.

Once a running job finishes, one of the misfired jobs will get picked up and run normally. However, Quartz seems to pick up a misfired job randomly, with no regard to the order the jobs had been originally scheduled to execute. Ideally, I'd like them to be picked up in the order they were supposed to have run, based on their original fire times.

Is it possible to make my waiting (misfired) jobs get fired in the order they were triggered once space in the Quartz ThreadPool becomes available?

Answer 1

When quartz handles a trigger that has missed it's fire time it will update the nextFireTime of the trigger. By default a trigger will be considered missed if it's nextFireTime is over 60 seconds in the past. Missed triggers should still be selected based on nextFireTime and priority order but I'm guessing it seems random because some triggers have been updated and others haven't.

I would suggest increasing the org.quartz.jobStore.misfireThreshold property. See http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigRAMJobStore.html (although the property is identical for all JobStores). This should make it less likely for your triggers to be re-scheduled.

Answer 2

In case of a CronTrigger, the method updateAfterMisfire() may re-schedule the task at new Date() case MISFIRE_INSTRUCTION_FIRE_ONCE_NOW policy.

If several task are misfired, several of them may be rescheduled at the same time (same millisecond), because computer runs fast.

As a consequence, and if no priority defined, the scheduler will pick-up the first next task, all with same NextFireTime, according to key or full name.

updateAfterMisfire() method should have re-schedule the task to a unique date using a Thread.sleep(25), as an example.

Answer 3

I sounds like you are running into a misfiring scenario (a scenario where there are more jobs ready to be executing than there are worker threads). Set the misfire instruction and/or priority property on the triggers to change how each behaves after it has past its fire time.

Also, you could consider increasing the misfire threshold, which would change the amount of time that a trigger can be "late" waiting for a thread to execute on before it is considered misfire (and has its misfire instruction applied to it).

Is it possible to make my waiting (misfired) jobs get fired in the order they were triggered once space in the Quartz ThreadPool becomes available?

The "do nothing" instructions will leave the fire times as-is.

Answer 4

Looking at the quartz thread pool , it uses a wait()/notify() loop, which is not fair, and will randomly select a new thread when multiple threads are waiting.

You could use your own instance of ThreadPool which is fair. Copy the code from SimpleThreadPool, but replace the locking around nextRunnableLock with a java.util.ReentrantLock, passing true to the fair constructor. In your modified SimpleThreadPool, use ReentrantLock.lock()/unlock() instead of synchronized, and use ReentrantLock.newCondition().signal()/await() instead of wait/notify, and it might solve your problem.