CODEWITHSUNDEEP
apache-camel
Luis Muñiz
Luis Muñiz

Reputation: 4811

Camel: How to synchronize competing imap consumers

I am looking to use apache-camel to poll an imap inbox, but I am wondering how this setup would behave in a cluster. I would deploy apache camel on each node of the cluster, and each node would poll the inbox.

How can I avoid having many consumers pick up the same message?

Upvotes: 1

Views: 1452

Answers (3)

Luis Muñiz
Luis Muñiz

Reputation: 4811

I decided to take the simple road and not install additional components. I used a clustered quartz job to trigger the polling of the inbox. The poller then places a retrieval command on a Hazelcast distributed queue, which is received by an array of Message retrieval components in the cluster.

Installing, Jms, James, in addition to Camel smelled to me, just to solve this task.

Upvotes: 3

Claus Ibsen
Claus Ibsen

Reputation: 55555

In a clustered environment you may consider having a way of electing a single Camel route that is active, which does the imap polling. And then have logic for failover if the node goes down.

In Camel you can take a look at route policy which can be applied to routes. http://camel.apache.org/routepolicy

The zookeeper component has a policy for electing a leader in a cluster, and only allow one route to be active. This requires though that you use zookeeper. http://camel.apache.org/zookeeper

Upvotes: 1

Petter Nordlander
Petter Nordlander

Reputation: 22279

Not very easy, since imap is not really a protocol for these kind of tasks.

The trick is still to have one consumer do the polling, not many. If you have many nodes for high availablility, you could do some tricks with JMS to trigger IMAP polls.

For instance, you could use a jms trigger message to init a poll and have all members of the cluster listen to that poll. Keep the concurrentConsumer to 1 and async. JMS disabled in Camel. You can rely on Message Groups or ActiveMQ exclusive consumer to be sure that only one node gets the trigger messages (when alive, otherwise another node will take over). Generating the polling messages might be tricky, but could be done as simply as a timer route from each camel node. Just tune the frequency.

This setup will avoid race conditions in IMAP, while not beeing load balanced, at least fail over secured. It might be good enough to just go ahead and do concurrent polling, with few issues. However, I don't think you will be 100% safe without only allowing one consumer.

Upvotes: 1

Related Questions