CODEWITHSUNDEEP
amazon-web-servicesamazon-sqs
rightCoder
rightCoder

Reputation: 301

Efficient way to check whether SQS queue is empty

I have a SQS Queue from which messages are read by multiple hosts. I want to run some job (business logic) after all the messages in the queue have been processed.

How can I check that the queue is empty?

Yes, I can check for ApproximateNumberOfMessages and ApproximateNumberOfMessagesNotVisible queue attributes but these are approximate numbers. I want to stop my hosts polling for messages in the queue when there are no messages left and then run the required job.

Any ideas? Thanks

Upvotes: 19

Views: 31390

Answers (3)

Corey
Corey

Reputation: 501

In case anyone else sees this: AWS documentation recommends checking that the number of messages in a queue is zero for several minutes straight before considering that queue empty. This is due to the distributed nature of SQS and because these metrics are approximate.

To confirm that a queue is empty (AWS CLI, AWS API)

  1. Stop all producers from sending messages.

  2. Repeatedly run one of the following commands:

  3. Observe the metrics for the following attributes:

    • ApproximateNumberOfMessagesDelayed
    • ApproximateNumberOfMessagesNotVisible
    • ApproximateNumberOfMessages

    When all of them are 0 for several minutes, the queue is empty.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html

You can have your hosts check the number of messages in the queue every few seconds for several minutes and keep track of how long the queue has been continuously empty. Once you confirm the queue is empty, you can run the required job.

Alternatively, if you are using a FIFO queue with a single message group and know beforehand which message from the producer will be the last one, you can add a message attribute that marks that message as the last one.

Upvotes: 0

Rohit
Rohit

Reputation: 927

You could trigger your post business logic on the cloud watch metric related to queue depth. When the depth is 0 then you can send a SNS notification or start a lambda function.

Also this cloud watch metric is better then others since this is actual message count reported by sqs service.

Upvotes: 4

Anthony Neace
Anthony Neace

Reputation: 26003

You could simply note empty receives from the API response while you're polling. Concerning CloudWatch, there is another metric that would be a better fit for this. From the documentation:

NumberOfEmptyReceives

The number of ReceiveMessage API calls that did not return a message.

Units: Count

Valid Statistics: Average, Minimum, Maximum, Sum, Data Samples (displays as Sample Count in the Amazon SQS console)

Some additional info:

  • This metric only gets populated every 5 minutes. If you set up an alarm based on this metric, this means your minimum period should be 5 minutes.
  • Sum is the most sensible statistic for your use case. If NumberOfEmptyReceives > 0, your polling job checked the queue and received no messages.

I personally used this metric to set up a cloudwatch alarm that will scale down an autoscaling group that hosts my polling job after the sum of NumberOfEmptyReceives > 0 for several consecutive periods. I like doing consecutive periods because it makes it more evident that the queue was not only empty, but has stayed empty.

Upvotes: 9

Related Questions