CODEWITHSUNDEEP
rabbitmqapache-camel
user3487839
user3487839

Reputation: 85

Camel-rabbitmq: Consuming from multiple rabbitmq queues in a single camel consumer

I have the following scenario: There are 3 rabbitmq queues to which producers push their messages based on the priority of the message.(myqueue_high, myqueue_medium, myqueue_low) I want to have a single consumer which can pull from these queues in order or priority i.e. it keeps pulling from high queue as long as messages are there. o/w it pulls from medium. If medium is also empty it pulls from low.

How do i achieve this? Do i need to write a custom component?

Upvotes: 1

Views: 2341

Answers (1)

Peter Keller
Peter Keller

Reputation: 7646

It would be easier to put all the messages to one queue but with different priorities. That way, the priority sorting would be done in the broker and the Camel consumer would get the messages already sorted by priority. However, RabbitMQ implements the FIFO principle and does not support priority handling (yet).

Solution 1

Camel allows you to reorganise messages based on some comparator using a Resequencer: https://camel.apache.org/resequencer.html:

from("rabbitmq://hostname[:port]/myqueue_high")
    .setHeader("priority", constant(9))
    .to("direct:messageProcessing");

from("rabbitmq://hostname[:port]/myqueue_medium")
    .setHeader("priority", constant(5))
    .to("direct:messageProcessing");

from("rabbitmq://hostname[:port]/myqueue_low")
    .setHeader("priority", constant(1))
    .to("direct:messageProcessing");

// sort by priority by allowing duplicates (message can have same priority)
// and use reverse ordering so 9 is first output (most important), and 0 is last
// (of course we could have set the priority the other way around, but this way
// we keep align with the JMS specification...)
// use batch mode and fire every 3th second
from("direct:messageProcessing")
    .resequence(header("priority")).batch().timeout(3000).allowDuplicates().reverse()
    .to("mock:result");

That way, all incoming messages are routed to the same sub route (direct:messageProcessing) where the messages are reordered according the priority header set by the incoming routes.

Solution 2

Use SEDA with a prioritization queue:

final PriorityBlockingQueueFactory<Exchange> priorityQueueFactory = new PriorityBlockingQueueFactory<Exchange>();
priorityQueueFactory.setComparator(new Comparator<Exchange>() {
    @Override
    public int compare(final Exchange exchange1, final Exchange exchange2) {
        final Integer prio1 = (Integer) exchange1.getIn().getHeader("priority");
        final Integer prio2 = (Integer) exchange2.getIn().getHeader("priority");
        return -prio1.compareTo(prio2);  // 9 has higher priority then 0
    }
});

final SimpleRegistry registry = new SimpleRegistry();
registry.put("priorityQueueFactory", priorityQueueFactory);

final ModelCamelContext context = new DefaultCamelContext(registry);
// configure and start your context here...

The route definition:

from("rabbitmq://hostname[:port]/myqueue_high")
    .setHeader("priority", constant(9))
    .to("seda:priority?queueFactory=#priorityQueueFactory");  // reference queue in registry

from("rabbitmq://hostname[:port]/myqueue_medium")
    .setHeader("priority", constant(5))
    .to("seda:priority?queueFactory=#priorityQueueFactory");

from("rabbitmq://hostname[:port]/myqueue_low")
    .setHeader("priority", constant(1))
    .to("seda:priority?queueFactory=#priorityQueueFactory");

from("seda:priority")
    .to("direct:messageProcessing");

Solution 3

Use JMS such as Camel's ActiveMQ component instead of SEDA if you need persistence in case of failures. Just forward the incoming messages from RabbitMQ to a JMS destination with setting the JMSPriority header.

Solution 4

Skip the RabbitMQ entirely and just use a JMS broker such as ActiveMQ that supports prioritization.

Upvotes: 4

Related Questions