CODEWITHSUNDEEP
pythonamazon-web-servicesbotoamazon-sqs
Ajay Nair
Ajay Nair

Reputation: 1867

Reading data consecutively in a AWS SQS queue

I am very new to AWS SQS queues and I am currently playing around with boto. I noticed that when I try to read a queue filled with messages in a while loop, I see that after 10-25 messages are read, the queue does not return any message (even though the queue has more than 1000+ messages). It starts populating another set of 10-25 messages after a few seconds or on stopping and restarting the the program. while true: read_queue() // connection is already established with the desired queue.

Any thoughts on this behaviour or point me in the right direction. Just reiterating I am just couple of days old to SQS !!

Thanks

Upvotes: 3

Views: 2132

Answers (2)

garnaat
garnaat

Reputation: 45846

Long polling is more efficient because it allows you to leave the HTTP connection open for a period of time while you wait for more results. However, you can still do your own polling in boto by just setting up a loop and waiting for some period of time between reading the queue. You can still get good overall throughput with this polling strategy.

Upvotes: 2

jeffknupp
jeffknupp

Reputation: 6254

That's the way that SQS queues work by default (short polling). If you haven't changed any settings after setting up your queue, the default is to get messages from a weighted random sampling of machines. If you're using more than one machine and want all the messages you can consume at that moment (across all machines), you need to use long polling. See the Amazon documentation here. I don't think boto supports that directly ATM.

Upvotes: 2

Related Questions