CODEWITHSUNDEEP
parallel-processingapache-camel
Nitin
Nitin

Reputation: 11

message ordering and parallel processing apache camel

I'd like to process a list of custom java objects , split those using camel splitter and like to process it in parallel threads. But the challenge I am facing is ,list of custom objects is ordered based on Id, which has to be written in a file.

As soon as I use parallel processing, the sequence is disturbed. I went through a few articles which asked to use "resequencer" or "single thread". But using single thread, it takes huge time to process 5k records.

Any leads would be highly helpful. Thanks Nitin

Upvotes: 1

Views: 1211

Answers (2)

ah.narayanan
ah.narayanan

Reputation: 195

You can create an instance of AggregationStrategy that compares the results of newExchange and oldExchange and create a resultExchange with sorted list of custom java objects based on id.

But using single thread, it takes huge time to process 5k records.

You have to be careful as you may not want to spin up 5k parallel threads but instead create your own thread pool attach it in the split with executorServiceRef. This way you can control the number of threads and handle what to do when your queue is full.

Upvotes: 1

Divya
Divya

Reputation: 61

I had similar kind of problem while Splitting a XMl request based on a tag "XXX". Then processing the splitted request and Aggregating into a Response. The order of the Aggregated response is not same as the request.

FIX : The issue has been resolved by using Aggregation "strategyRef" in splitter EIP.

Sample Code:

    <route>
        <from id="_from1" uri="activemq:DocumentGenerationQueue" />
        <split  parallelProcessing="true" streaming="false"  strategyRef = "AggregateTask" >
        <tokenize token="XXX" xml="true" />
            <to id="_to71"  uri="bean:ProcessorBean" />
            <to id="_to72"  uri="activemq:SplittedResponseQueue" />
        </split>
        <to uri="activemq:AggregatedResponseQueue" />
    </route>

Upvotes: 5

Related Questions