CODEWITHSUNDEEP
javaspringspring-cloudamazon-sqs
Murilo Locatelli
Murilo Locatelli

Reputation: 178

Amount parallel processing Simple Queue Service (SQS)

I am using Spring Cloud to consume Simple Queue Service (SQS). I have the following configurations for parallel processing:

@Bean
public SimpleAsyncTaskExecutor simpleAsyncTaskExecutor() {
    SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
    simpleAsyncTaskExecutor.setConcurrencyLimit(50);
    return simpleAsyncTaskExecutor;
}

@Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory(
        SimpleAsyncTaskExecutor simpleAsyncTaskExecutor) {

    SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
    factory.setAutoStartup(true);
    factory.setTaskExecutor(simpleAsyncTaskExecutor);
    factory.setWaitTimeOut(20);
    factory.setMaxNumberOfMessages(10);

    return factory;
}

I need to process 50 messages in 50 threads (configuration in the bean SimpleAsyncTaskExecutor), but is processing only 10 messages in parallel (maxNumberOfMessages returned from SQS)

How can I process 50 messages instead 10?

Upvotes: 2

Views: 6813

Answers (2)

Murilo Locatelli
Murilo Locatelli

Reputation: 178

I found the solution.

It's necessary to annotate the method with @Async, change deletionPolicy to NEVER, and delete the message when finalizing execution.

In this way, the queue consume will respect the configured number of threads. For example, if you have 50 threads, will make 5 requests in the SQS queue (10 messages per request), thus processing a total of 50 messages in parallel.

The code looks like this:

@Async
@SqsListener(value = "sqsName", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
public void consume(String message, Acknowledgment acknowledgment) throws InterruptedException, ExecutionException {

    //your code

    acknowledgment.acknowledge().get(); //To delete message from queue
}

Upvotes: 5

Martin Hansen
Martin Hansen

Reputation: 2101

I wouldn't be into specific numbers (like 50 messages for 50 threads) too much. Try performance testing it instead (build something to push the expected number of messages in peak-hours to the queue, and let your service handle them, to see if it bottlenecks).

As per your actual question, you can't. AWS SQS simply doesnt support fetching more than 10 messages pr. request. see http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html for reference. (it's in the 1st paragraph).

Upvotes: 0

Related Questions