CODEWITHSUNDEEP
javamultithreadingjmsactivemq-classic
user1411957
user1411957

Reputation: 21

How to process messages in the same order from queue using multiple servers in java

We have a ActiveMQ queue which will receive 100k stock order messages(each message contains stock name, sell price, bid price in json format) per second. Out of 100k messages/sec there can be n no.of messages of single stock. If we receive multiple messages of same stock then we need to process all those messages in the same order using java. We can't process 100k messages/second using single listener in one server. Need to process it by using multiple listeners & servers but display the result in UI using the same order that is placed in Queue.

Read Stock Queue--> Validate the request -->Update the Stock price in UI

Example message:- { stockName:"TCS", sellPrice:"102", bidPrice:"100" }

Can you suggest solution for the above problem.

Upvotes: 0

Views: 1212

Answers (3)

Reddy_73
Reddy_73

Reputation: 134

We had a similar requirement and we used an open source framework called LMAX Disruptor, supposedly highly performant concurrency framework. You can experiment around it, https://github.com/LMAX-Exchange/disruptor/wiki/Getting-Started.

On a very high level:

  1. Put the Stocks received into a ringbuffer [core data structure that the framework is built upon], this would be the consumer for ActiveMQ and producer for the ringbuffer.

  2. The consumers/workers[in your case multiple - mulltiple here is a worker-thread for each unique stock-name] pick up the Stocks from ringbuffer in ordered fashion. In the worker/listener, you can handle event based on condition.

I've just committed sample code trying to demonstrate your use case, for your reference: https://github.com/reddy73/Disruptor-Example

Upvotes: 0

Axel Podehl
Axel Podehl

Reputation: 4303

I would suggest to use non-persistent publication to topics instead of queues. Topics give you the flexibility of

  1. choosing subscription wildcards and
  2. possibly adding other services later to this architecture. You might not require this now, but maybe in 5 years you will need another GUI or some form of monitoring or replay service. In case you used topics you can just plugin new subscribers - you don't have to change your publication side for those...

You can use durable subscriptions if you need more persistence.

Message order is guaranteed within the same publication topic so you should make the stock name part of the topic. You could publish on something like ORDER.STOCK.TCS.

But having a balanced load based on stock names is tricky because some letters like Z are very rare, while others are frequent. So in addition to the stock name add the stock name's hash%100 to the topic. For Example if the hashcode of TCS is 12357 and you do modulo 100, you publish this on ORDER.STOCK.TCS.57

Let's say you have 10 subscriber's, each subscriber could then make 10 subscriptions. For example subscriber 1 would subscribe to ORDER.STOCK.*.0, ORDER.STOCK.*.1, ... ORDER.STOCK.*.9

Subscriber 2 would subscribe to ORDER.STOCK.*.10, ORDER.STOCK.*.11, ... ORDER.STOCK.*.19

If you have 5 subscribers, each one does 20 subscriptions (you get the idea). The reason for this is that

Upvotes: 0

Sharon Ben Asher
Sharon Ben Asher

Reputation: 14328

Here is my proposal:
You need to split the queue to sub queues based on the stock name. you can split based on the first letter(s) of the stock name. this will give you ample parallel capabilities while ensuring that all messages of the same stock land on one queue. there will need to be one reader from the main queue but all it does is forwarding the messages to the sub queues.

Upvotes: 1

Related Questions