CODEWITHSUNDEEP
amazon-elastic-beanstalkamazon-sqsamazon-sns
northernMonkey
northernMonkey

Reputation: 1393

Amazon SQS Worker Tier Auto Scaling fails

We have an SQS Worker Tier app subscribed to a queue. When it is running it works fine, however, when it gets busy, and scales up, the new instance starts getting messages almost immediately, before it is actually ready. This results in 500 responses, and the messages being discarded to the dead letter queue.

We have our queue configured with a maximum attempt of 1; due to the database changes a message will make during consumption we can't just put it back in the queue in case of error.

I have tried using the monitor health url as I would with a normal web app, but this doesn't seem to work as messages continue to be sent regardless.

Is there any way of setting a delay on any new auto scaled instance before is starts receiving messages from the queue?

Upvotes: 1

Views: 558

Answers (2)

flovilmart
flovilmart

Reputation: 1769

You can set the HTTP Connection setting (Configuration > Worker Configuration) in order to limit the number of concurrent connections to your worker. If you set it to 1, you're sure that 1 worker won't receive another request unless it has already responded.

Upvotes: 1

E.J. Brennan
E.J. Brennan

Reputation: 46849

I am not sure how the instance is 'getting messages' before its ready, unless you are actually using SNS to PUSH the messages to the endpoint, as opposed to having the endpoint(instance) PULL the messages from the queue.

If you are pushing messages via SNS, then the easiest solution is to have the instance POLL the SQS queue for messages when its ready to process them - much safer and reliable, and obviously the instance can decide for itself when its ready to do work.

It also sounds to me like you solution is not architected properly. If accidentally processing the same message twice would cause you problems in your database, then your not using SQS in the correct manner. Work that SQS does should be idempotent - i.e. it should be able to processed more than one time without causing problems. Even when everything is running 100% correctly, on your end and at AWS, its possible that the same message will be sent more than once to your workers - you can't prevent that - and your processing needs to be able to handle that gracefully.

Upvotes: 1

Related Questions