CODEWITHSUNDEEP
amazon-web-servicesqueuemessage-queueamazon-sqs
Fizban
Fizban

Reputation: 661

Throttling SQS to have a maximum of 1 in-flight message cross all subscribers?

I have a situation where I am triggering a long running process on a microservice by a sqs message. I have multiple instances of the same microservice but I only want one instances to run the task. Is there a way to specify that the queue should only deliver 1 message and wait until the subscriber respond or it reach the visibility timeout?

Thanks

Upvotes: 1

Views: 1296

Answers (2)

qkhanhpro
qkhanhpro

Reputation: 5220

I want to send one message only to subscriberA and wait until subscriberA process the message completely or until the timeout. I cant see how to avoid the subscriberB to pickup another message from the queue until subscriberA finish

It's strange that you have an architecture like so. That will make use of one subscriber at a time so a fleet of subscriber / multi-instance microservice is not appropriate.

If it is possible, please just let one of your microservices subscribe/poll the queue

Otherwise, you will have to it on your own. My suggestion:

  • Have two queue. One of the queue would receive all messages. One of the queue would serve the message to your subscribers. Your code base won't have to change much if It's set-up like this
  • Set up a Lambda function triggered by Receiving-SQS. Set the concurrency of that function to 1, adjust the delivery delay of your SQS to throttle as you wish
  • The function will need to check the Serving-SQS whether there is message in that queue ( Waitting or being processed) through the GetQueueAttributes API
  • Send the message the Serving-SQS if it's free or return the message back to Receiving-SQS

Upvotes: 1

DominikHelps
DominikHelps

Reputation: 1021

SQS already does this for you. The attribute "visibility timeout" ensures that the task is only visible to 1 worker if you align it with the maximum execution time.

The worker will fetch the object from SQS. SQS ensures that it is only delivered once and invisible to others in the time of the visibility timeout. You should give a maximum execution time for the task. Lets say 5 minutes. And put a higher visibility timeout. Lets say 6 minutes. After this time if the worker did not successfully executed the task the item will be invisible to others for another minute and then return to the queue. After this another worker can pick up the message and it is invisible to others for 6 minutes again. On success the worker should remove the message.

This way you ensure that every item gets only processed ones.

Upvotes: 0

Related Questions