CODEWITHSUNDEEP
javamultithreadingqueuejmsmessage-queue
user1172468
user1172468

Reputation: 5464

Queuing / Worker Thread architecture for a single java process

I have the following problem to solve.

I need to write a java program that:

  1. reads JSON object j1,j2,...,jn from a web service.
  2. does some number crunching on each object to come up with j1',j2',...,jn'
  3. Sends objects j1',j2',...,jn' to a web service.

The computational, space requirements for steps 1,2, and 3 can vary at any given time.

For example:

  1. The time it takes to process a JSON object at step 2 can vary depending on the contents of the JSON Object.
  2. The rate of objects being produced by the webservice in step 1 can go up or down with time.
  3. The consuming web service in step 3 can get backlogged.

To address the above design concerns want to implement the following architecture:

enter image description here

  1. Read JSON objects from the external webservice and place them on a Q
  2. An automatically size adjusting worker thread pool that consumes JSON objects from the Q and processes them. After processing them, places the resulting objects on the second Q
  3. An automatically size adjusting worker thread pool that consumes JSON objects from the second Q to send them to the consuming webservice.

Question:

I am curious if there is framework which I can use to solve this problem?

Notes:

  1. I can solve this using a range of components like custom Queues, Threadpools using the concurrency package -- however I'm looking for a solution that allows the writing of such a solution.
  2. This is not going to live inside a container. This will be a Java process where the entry point is public static void main(String args[])
  3. However if there is a container suited to this paradigm I would like to learn about it.
  4. I could split this into multiple processes, however I'd like to keep it very simple and in a single process.

Thanks.

Thanks.

Upvotes: 1

Views: 1148

Answers (2)

mzzzzb
mzzzzb

Reputation: 1452

try Apache camel or Spring Integration to wire things up. These are kind of integration frameworks, will ease your interaction with webservices. What you need to do is define a route from webservice 1 -> number cruncher -> web service 2. routing and conversion required in between can be handled by the framework itself

you'd implement your cruncher as a camel processor. parallelizing your cruncher may be achieved via SEDA; Camel has a component for this pattern. Another alternate would be AsyncProcessor

I'd say you first take a look at the principles behind frameworks like camel. The abstractions they create are very relevant for the problem in hand.

Upvotes: 1

Jeff Richley
Jeff Richley

Reputation: 1

I'm not exactly sure what the end question is for your post, but you have a reasonable design concept. One question I have for you is what environment are you in? Are you in a JavaEE container or just a simple standalone application?

If you are in a container, it would make more sense to have Message Driven Beans processing off of the JMS queues than having a pool of worker threads.

If in your own container, it would make more sense for you to manage the thread pool yourself. With that said, I would also consider having separate applications running that pull the work off of the queues which would lead to a better scaling architecture for you. If the need ever came up, you could add more machines with more workers pointing at the one queue.

Upvotes: 1

Related Questions