CODEWITHSUNDEEP
.netqueueamazon-sqsamazon-sns
Pietro
Pietro

Reputation: 781

Design a push queue using Amazon SQS and SNS, how to?

Right now:

As App #2 could take quite long to process the messages and I cannot forecast how many messages are sent simultaneously from App #1, the queue system guarantees me that App #2 doesn't run out of resources and the system can scale quite easily. But there's a problem I'd like to solve: I would prefer not to have a machine to run the worker role (now on Azure) just to poll a queue (everything is hosted elsewhere where worker role is not an option). Moreover polling won't ever be so responsive as push because of pauses between polls.

Switching from poll to push sounds the right path but I need I need to keep the guarantee that even if 1k messages are sent from App #1 in one second, App #2 process them one by one and is not hit 1k times per second.

I was planning a design where App #1 publish to SNS Topic whose subscribers are SQS and App #2. App #2 checks the SQS queue and if it is empty it quits, if not process the messages one by one and then quits. But how can I code App #2 (right now a .Net web/web service) so that if it is already processing a message and receive a notification from SNS it does nothing and quits (if not multiple processing would be run).

Any suggestions how to design this? I read this blog post but I do not know how to avoid the processing app to process multiple messages simultaneously.

Upvotes: 2

Views: 1780

Answers (2)

Andrew Hancox
Andrew Hancox

Reputation: 2340

You can get round all of this by using the delivery policies, these allow you to specify rate limits and retry rules for your endpoints.

Upvotes: 0

Chen Harel
Chen Harel

Reputation: 10062

If you think of the SNS role as a way to shorten the polling "sleep intervals" then you simply don't change any of the query power, app #2 will still process the messages as it pleases but if it "sleeps" the notification will awaken it and fire a poll right away.

Upvotes: 2

Related Questions