CODEWITHSUNDEEP
amazon-web-servicesamazon-sqsautoscalingamazon-sns
Thomas S.
Thomas S.

Reputation: 1

Build flexible publish/subscribe pattern with AWS Services

I want to synchronize messages between several application instances on amazon beanstalk. Every instance must be able to send messages to all other instances without knowing them. So I thought I build a publish/subscribe pattern with SNS and SQS.

When a new application instance is starting, it creates a new SQS queue and subscribes to the topic. So far, so good. But what if the autoscaler shuts down an instance?

Is there any possibility to catch the shutdown event? Or do you have any Ideas for other implementations?

Upvotes: 0

Views: 679

Answers (2)

badera
badera

Reputation: 1545

This post is old, but I just run into it and have had similar questions this week - since the post is still not marked as accepted, I give a try.

I see two solutions:

1) If the rentention of a message is longer than the period you want to check for "unused" queues: You can use the API to retrieve a list of all queues and check the ApproximateNumberOfMessages. If it is about a specified treshold, you can assume that the queue is dead and you can delete it. Either you are sure to have always the specified amount of messages sent in the given period or you need to define a kind of "keep alive" message, which has only the goal to fill up the dead queues to your threshold.

2) If it is possible to detect how many / which nodes are running - e.g. let each node periodically (short period, executed on every node) update a DB entry with its nodeName / lastActivity timestamp and then periodically (long period, executed on one node only) retrieve this list -> go through every queue (retrieved by API) and check whether the corresponding node is still active and if not, delete the queue.

Upvotes: 0

Anthony Neace
Anthony Neace

Reputation: 26003

You sound like you would be well-served by existing metrics that you can set alarms with via CloudWatch, or by making your own custom metrics. For example, each EC2 instance has several metrics to monitor recurring Status Checks. You can define alarms that will send out notifications to subscribers based on their state. An instance that fails a Status Check for more than X minutes may be worth investigating... which could happen automatically if you have an application monitoring alarms.

If that isn't quite narrow enough for you, I would encourage you to check out custom metrics. You could create your own metrics and alarm to (for example) ping your applications per-instance to make sure they're responsive before synchronizing the messages. Alarms have three states: "OK", "ALARM", and "INSUFFICIENT DATA" that you can listen for and respond appropriately. So for example, if your metric polled on the same instance as the application and the instance went down, you would start getting "INSUFFICIENT DATA" messages.

Custom metrics have been helpful for me before as a simple application monitor... report a 1 or a 0 if I can ping my application every so often. Send a notification if the ping fails, send a notification if there is insufficient data. Worked great.

Upvotes: 1

Related Questions