CODEWITHSUNDEEP
parallel-processingapache-camelamazon-sqs
Jeffrey Licht
Jeffrey Licht

Reputation: 17

How can I use multiple cores to process messages from SQS with Camel

I am using Camel to read messages off an Amazon SQS queue, do some CPU-intensive processing, and then place them on another SQS queue. Camel is being invoked through the maven plugin, using mvn camel:run.

When run on a server with multiple cores, Camel only appears to be using a single core to process messages (as observed by monitoring CPU utilization of all cores). How can I take advantage of all available cores?

Things I have tried:

The route in question:

     <route id="marctomods"  errorHandlerRef="eh">
        <from uri="aws-sqs://{{sqs.environment}}-normalize-marcxml?accessKey=${access.key}&amp;secretKey=${secret.key}&amp;amazonSQSClient=#sqsClient&amp;maxMessagesPerPoll=10" />
        <process ref="modsProcessor"/>
        <to uri="aws-sqs://{{sqs.environment}}-enrich?accessKey=${access.key}&amp;secretKey=${secret.key}&amp;amazonSQSClient=#sqsClient" />
     </route>

Upvotes: 1

Views: 1183

Answers (1)

matt helliwell
matt helliwell

Reputation: 2678

You generally only get a single message in transit through a route. http://camel.apache.org/parallel-processing-and-ordering.html explains the parallel processing options.

You could, for example, read messages from sqs onto a seda queue and then have multiple parallel consumers reading from the seda queue, eg

<route>
    <from uri="aws-sqs://{{sqs.environment}}-normalize-marcxml />
    <to uri="seda:myQueue">
</route>

<route>
    <from uri="seda:myQueue?concurrentConsumers=10" />
    <process ref="modsProcessor"/>
    <to uri="aws-sqs://{{sqs.environment}}-enrich>
</route>

Upvotes: 0

Related Questions