java ee job scheduling and execution

Question

In a JEE application there is a need to get rid of numerous JMS queues that support workflow execution. A workflow typically consists of multiple sequential tasks. Tasks may contain external system calls and other business logic. Task processors are implemented as MDB. When a task processor receives a message from the input queue it performs the business logic, enriches the input message and puts it to the input queue of the next task processor in the workflows chain. If a task processor failes to perform the task depending on settings the input message is rolled back in the queue for delayed execution or is put in the error queue. Now, instead of the JMS queues a database should be used. Tasks should be persisted as rows in a database table including task status (delayed, in progress, failed and so on), destination (name/alias of the processor which should execute the task) and of course business data.

I already started to implement this requirement using JCA. My inbound connector scans a DB table for tasks and then calls the apropriate endpoints (MDBs that implement a business interface).

Are there other ways to satisfy the requirements? Maybe somebody know a lightweight open source framework for such scenarios? Please do not mention BPM frameworks.

Answer 1

I assume that you want to get rid of JMS because managing a lot of messages in a lot of queues is not as "lightweight" as you want... and your code base is growing!

• You are constantly monitoring many queues to see where problems or bottlenecks are.
• If one of the external systems is down for a while, many calls are made and fail and all the messages run into the error queue.
• If you want to find out what happened to a single instance of your "workflow", you're grep'ing the log files.
• If you want to see the workflow from end to end, you'll have to resort to hand written documentation (which is not always accurate).
• If you add or remove one step, you'll have to find and update all steps that lead there, although they didn't change.
• Before you can deploy a new version, you're waiting for the jobs with the old version to run to a final state.

And while your JMS solution has fulfilled them, there are many more non-functional requirements to consider for new solutions: scalability, reliability, etc. All in all, those are tough requirements to fulfill... with any technology.

Maybe you can consider different areas where "lightweight" is most important to you:

• Can you test the plain business logic parts without starting the entire framework? I.e. how strongly is the business logic coupled to the framework?
• How difficult is it to learn to program and operate the system? How difficult is it to find people who already know?
• Can you migrate from the old system to the new step-by-step or only in a big bang?

There are numerous solutions in this area, but none of them is a one-stop out-of-the-box solution for all problems. Some are batch oriented (like Spring Batch or Java Batch), others are message oriented (like Akka or Hystrix). Comparing them is a science in its own right. There are so many forces for or agains them, you won't get a perfect answer here, even if you would provide a lot of detail, like peak number of jobs, number of steps, number of external system calls, etc. I've seen it again and again: If you exactly know the business workflows, the technology can be replaced more easily... so that's a prerequisite.

Then I'd advise you to play around with different solutions (if you have the time), or stay where you are... it's probably not the worst of all possible solutions. Don't... and I really mean it: Don't implement yet another "lightweight" thing, which you can bet, won't be very "lightweight" much sooner than you can imagine.

One last thing: I share your feelings about traditional BPM frameworks, namely BPEL and friends. But there are lightweight and open-source frameworks like Camunda BPM that are worth to take a look at.