CODEWITHSUNDEEP
javadesign-patternsqueuerabbitmq
user2610972
user2610972

Reputation: 31

rabbitMQ consume from 2 queues

I have to write heavy load system, with pretty easy task to do. So i decided to split this tasks into multiple workers in different locations (or clouds). To communicate i want to use rabbitmq queue.

In my system there will be two kinds of software nodes: schedulers and workers. Schedulers will take user input from queue_input, split it into smaller task and put this smaller task into workers_queue. Workers reads this queue and 'do the thing'. I used round-robbin load balancing here - and all works pretty well, as long, as some worker crashed. Then i loose information about task completion (it's not allowed to do single operation twice, each task contains a pack of 50 iterations of doing worker-code with diffirent data).

I consider something like technical_queue - another channel to scheduler-worker communication, and I wonder, how to design it in a good way. I used tutorials from rabbitmq page, so my worker thread looks like :

while(true) {
   message = consume(QUEUE,...);
   handle(message); //do 50 simple tasks in loop for data in message
}

How can i handle second queue? Another thread we some while(true) {} loop?, or is there a better sollution to this? Maybe should I reuse existing queue with topic exchange? (but i wanted to have independent way of communication, while handling the task, which may take some time.

Upvotes: 3

Views: 1574

Answers (2)

Walter Caraza
Walter Caraza

Reputation: 911

I would recommend you create a second queue consumer

consumer1 -> queue_process
consumer2 -> queue_process

Both consumers should make listening to the same queue. Greetings I hope will help

Upvotes: 0

Adam Gent
Adam Gent

Reputation: 49085

You should probably take a look at (doc). I hate to tell you to add a layer but that spring library takes care of the threading issues and management of threads with its SimpleMessageListenerContainer. Each container goes to a queue and you can specify # of threads (ie workers) per queue.

Alternatively you can make your own using an ExecutorService but you will probably end up rewriting what SimpleMessageListenerContainer does. Also you just could execute (via OS or batch scripts) more processes and that will add more consumers to each queue.

As far as queue topology is concerned it is entirely dependent on business logic/concerns and generally less on performance needs. More often you had more queues for business reasons and more workers for performance reasons but if a queue gets backed up with the same type of message considering giving that type of message its own queue. What your describing sounds like two queues with multiple consumer on your worker queue.

Other than the threading issue and queue topology I'm not entirely sure what else you are asking.

Upvotes: 1

Related Questions