CODEWITHSUNDEEP
amazon-web-servicesamazon-sqs
ShaneKm
ShaneKm

Reputation: 21308

AWS SQS Receive Messages -- How to Know when Queue is Empty

I want to get all the messages in the queue to process them. However the property for MaxNumberOfMessages is 10 (based on documentation)

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

How can I read in all messages so I can process them? Or how would I know when queue is empty?

thanks

Upvotes: 4

Views: 7930

Answers (2)

Corey
Corey

Reputation: 501

To check if the queue is empty you have to verify the total number of messages in the queue is zero. SQS does not provide a single metric for this, rather you have to calculate the sum of three different metrics.

From the docs:

To confirm that a queue is empty (AWS CLI, AWS API)

  1. Stop all producers from sending messages.

  2. Repeatedly run one of the following commands:

  3. Observe the metrics for the following attributes:

    • ApproximateNumberOfMessagesDelayed
    • ApproximateNumberOfMessagesNotVisible
    • ApproximateNumberOfMessages

    When all of them are 0 for several minutes, the queue is empty.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html

Getting an empty response from a ReceiveMessage call does NOT necessarily mean the queue is empty. You can have messages in the queue and still receive an empty response if:

  1. Messages are delayed - You can set delays on individual messages for standard queues or at the queue level for standard and FIFO queues. During the delay period messages are invisible to consumers.
  2. Messages are in-flight - When a consumer receives a message, that message remains in the queue until the consumer deletes it by calling DeleteMessage. While the message is in this state it is considered in-flight and is not available for other consumers.
  3. Multiple messages have the same message group id in a FIFO queue - When a consumer receives a message from a FIFO queue, no other consumer can receive messages from the same message group. This ensures messages are processed in FIFO order.

By summing the metrics listed above, you can account for all of these scenarios.

Upvotes: 1

Matt Morgan
Matt Morgan

Reputation: 5303

When you receive messages from the queue, they are marked as "in flight." After you successfully process them, you send a call to the queue to delete them. This call will include IDs of each of the messages.

When the queue is empty, the next read will have an empty Messages array.

Usually when I do this I wrap my call to read the queue in a loop (a while loop) and only keep processing if I have Messages after doing a read.

It shouldn't make any difference if it's a FIFO queue or a standard one.

Upvotes: 6

Related Questions