How to improve message processing with large queues?

Question

I am trying to use Spring JMS and ActiveMQ to process a large number of messages. The context of the problem is the following: Each customer produces a set of messages that are added to the queue. The messages are added to the queue with the customer id as parameter.

In one case, customer A can add 10k messages to the queue, while customer B only adds 100 messages to that same queue. My issue is customer B needs to wait until all the 10k messages have finished processing before its 100 messages are processed.

Is there a way to process some of messages of customer A and some of the messages of customer B at the same time? I know there is the option to set a higher priority on the messages from customer B, but that does not solve the issue when there is more multiple customers. The customer with the more messages will fill-up the queue while the others will have to wait.

I would appreciate if you could provide some help or advice.

Answer 1

I think I have find the solution to the issue. It involves using message groups. For each message I set the property JMSXGroupID with an identifier for the customer.

Since they are multiple message groups the queue takes care of assigning messages from different groups to different consumers. In that way, documents from customer B can be processed while the ones from customer A are still being processed.

Answer 2

I would say you can fine tune your activemq the number of messages it is processing in batches. Also, there are way you can fine tune a given Broker and queue. For more details refer this link:

http://activemq.apache.org/performance-tuning

https://access.redhat.com/documentation/en-US/Fuse_ESB/4.4.1/html-single/ActiveMQ_Tuning_Guide/index.html

Answer 3

The basic semantic of a queue is first-in-first-out (i.e. FIFO). There's no real way to escape that. I recommend you redesign your application to use multiple queues - one for each "type" of message or independent application you have.