CODEWITHSUNDEEP
javaapache-camelcluster-computingquartz-schedulerhigh-availability
AdjustingForInflation
AdjustingForInflation

Reputation: 1611

How to use Camel Quartz clustering in tandem with queue consumption?

Using Apache 2.11.0 here. I am writing a little Camel-based app (using Spring XML) that consumes messages off a queue (inputQueue), aggregates them, processes the aggregated batch, and then sends out a generated report. I'd like to deploy this app (widget-reporter.war) to multiple Tomcat instances and cluster them so that if one node goes down, it is still running on other nodes and the report will still get generated.

So far the route's pseudo-code is as follows:

<camelContext id="myCamel" xmlns="http://camel.apache.org/schema/spring">
    <route id="myRoute">
        <from uri="activemq:queue:inputQueue" />
        <aggregate strategyRef="myStrategy" completionSize="500">
            <correlationExpression>
                <xpath>/fizz/buzz</xpath>
            </correlationExpression>
            <to uri="bean:reportGenerator?method=runReport"/>
        </aggregate>
    </route>
</camelContext>

So, as you can see, we dequeue messages off of inputQueue, send them to an aggregator (which correlates them on an xpath expression), and after we reach 500 messages we send the aggregated exchange off to a reportGenerator.

I have this working locally (1 node) perfectly. Now I'm trying to deploy this app to multiple Tomcat instances, but I only want it running on 1 node at a time. In other words, if I have this app deployed to 5 Tomcat instances, I don't want 5 Tomcats all consuming from inputQueue at the same time: only 1 should be operating at any given time, and the other 4 should be idling, waiting for the primary to failover to them.

It looks like Camel Quartz allows for such clustering, but the problem is that it requires me to start my route with a quartz:// endpoint, and not activemq. So I'd have to have something like this (which doesn't work):

<route id="myRoute">
    <from uri="quartz://runWidgetReporterExclusively?cron=0+15,45+0-16+?+*+SAT" />

    <!-- Can't have 2 consecutive "from" endpoints... -->
    <from uri="activemq:queue:inputQueue" />

    <aggregate strategyRef="myStrategy" completionSize="500">
        <correlationExpression>
            <xpath>/fizz/buzz</xpath>
        </correlationExpression>
        <to uri="bean:reportGenerator?method=runReport"/>
    </aggregate>
</route>

Does anybody know how I can either:

Upvotes: 3

Views: 2038

Answers (2)

Ben ODay
Ben ODay

Reputation: 21015

use a camel-zookeeper enabled RoutePolicy for this...

ZooKeeper allows for very simple and effective leader election out of the box; This component exploits this election capability in a RoutePolicy to control when and how routes are enabled. This policy would typically be used in fail-over scenarios, to control identical instances of a route across a cluster of Camel based servers. A very common scenario is a simple 'Master-Slave' setup where there are multiple instances of a route distributed across a cluster but only one of them, that of the master, should be running at a time. If the master fails, a new master should be elected from the available slaves and the route in this new master should be started.

Upvotes: 2

Ralf
Ralf

Reputation: 6853

I have never done this myself but...

If you drop your requirement for passive fallback nodes (they anyways seem a waste of resources to me) you could use the camel load balancer with a sticky balancing policy. The policy expects an expression to correlate the messages belonging to a particular "session". You should be able to re-use your xpath expression from the aggregator in your route for that.

Your example route does not indicate a requirement as to why you needed to run Camel in Tomcat. If your Camel application does not directly use resources that are managed by Tomcat, I would recommend to run your Camel application stand-alone; or to decouple it from the web-application to achieve it. Camel runs completely stand-alone, use the org.apache.camel.spring.Main class to fire up your spring and camel contexts.

Upvotes: 0

Related Questions