Persistent delayed jobs queue for Java

Question

I'm looking for an existing system to replace an existing slow and complicated self written mechanism of job management. The existing system:

• 1 MySQL DB with a long massive table of jobs - the queue
• Multiple servers (written in java) all extracting jobs from the queue and processing them
• a job might NOT be deleted from the queue after processing it, to rerun it later
• a job might create other jobs and insert them to the queue

The limitations:

• As more and more jobs are created and inserted in to the queue, it takes longer to extract jobs from it. (The jobs are chosen by priority and type) - create a bottle neck

I'm looking for an existing system that can replace this one, and improve it's performance. Any suggestions?

Thanks

Answer 1

You could:

• submit your jobs to Amazon's Simple Queue Service (maybe JAXB marshalled)
• dynamically start some EC2 instances according to your queue's length and probably
• submit the results (or availability notice for some files on S3) to Simple Notification Service (again JAXB marshalled).

That exactly what we do, using EC2 Spot instances to minimize costs. And that's what I call serious cloud computing ;)

Answer 2

I don't generally recommend JMS, but it sounds like it really is what you need here. Distributed, transactional, persistent job queue management is what JMS is all about.

Popular open-source implementations include HornetQ and ActiveMQ.