CODEWITHSUNDEEP
javaamazon-web-servicesevent-handlingamazon-sqsamazon-sns
northernMonkey
northernMonkey

Reputation: 1393

Creating Amazon SNS messages to be processed in the future

For the last few years we have used our own RM Application to process events related to our applications. This works by polling a database table every few minutes, looking for any rows that have a due date before now, and have not been processed yet.

We are currently making the transition to SNS, with SQS Worker tiers processing them. The problem with this approach is that we can't future date our messages. Our applications sometimes have events that we don't want to process until a week later.

Are there any design approaches, alternative services, clever tricks we could employ that would allow us to do achieve this?

One solution would be to keep our existing application running, at a simplified level, so all it does is send the SNS notifications when they are due, but the aim of this project is to try and do away with our existing app.

Upvotes: 8

Views: 17140

Answers (3)

Vladimir Venegas
Vladimir Venegas

Reputation: 4183

I found this approach: https://github.com/alestic/aws-sns-delayed. Basically, you can use a step function with a wait step in there

Upvotes: 1

shadowfax
shadowfax

Reputation: 981

A week might be too long as SQS message retention itself is only 15 days. If you are okay with maximum retention of 15days, one idea is to keep the changing the visibility of a message every time you receive until it is ready for processing. The maximum allowed visibility timeout is 12 hours. More on visibility timeout and APIs for changing them,

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html

Upvotes: 1

John Rotenstein
John Rotenstein

Reputation: 269081

The database approach would be the wisest, being careful that each row is only processed once.

Amazon Simple Notification Service (SNS) is designed to send notifications immediately. There is no functionality for a delayed send (although some notification types are retried if they fail).

Amazon Simple Queue Service (SQS) does have a delay feature, but only up to 15 minutes -- this is useful if you need to do some work before the message is processed, such as copying related data to Amazon S3.

Given that your requirement is to wait until some future arbitrary time (effectively like a scheduling system), you could either start a process and tell it to sleep for a certain amount of time (a bad idea in case systems are restarted), or continue your approach of polling from a database.

If all jobs are scheduled for a distant future (eg at least one hour away), you theoretically only need to poll the database once an hour to retrieve the earliest scheduled time.

Upvotes: 4

Related Questions