CODEWITHSUNDEEP
amazon-web-servicesamazon-sqs
Brandon
Brandon

Reputation: 7966

How can I freeze an SQS queue against any reads?

I have an SQS queue (standard, not FIFO) I'm using as a task queue for multiple workers on a long-running process.

I'm trying to add a 'kill switch' that'll temporarily stop my workers from being able to receive/process any messages.

I don't want to delete the messages or stop any new messages from entering the queue, just stop any readers until I toggle the kill switch again.

Is this possible to do with the SDK? or with permissions?

Upvotes: 4

Views: 13504

Answers (4)

erik258
erik258

Reputation: 16305

I dont know of an sqs pause button, and I don't see any sqs API calls that seem promising.

You could remove access from the iam policies allowing the readers to read.

You could remove access to sqs public ips at a network level.

But I hate those solutions; they blur failure scenarios with success and aren't really any more elegant or convenient than orchestrating within the application.

I'd advise my engineers to use a feature flag for this, in dynamo, consul, or some other consistent key value store. Makes a lot more sense to me than allowing all the reads to fail continuously while the permission or net access is removed

Upvotes: 7

Prashant Gupta
Prashant Gupta

Reputation: 1

Use the delivery delay setting.

On the AWS console you can go to edit the configuration of the queue and introduce a delay of upto 15 minutes in delivering the messages to queue. This way consumers will not see the message for that duration.

This way you can get 15 minutes to do whatever cleanup you want. But if its going to take longer than that, then you will need a build a kill switch.

We used this to buy time for our nodes to be restarted/redeployed without impacting publishers.

Upvotes: -1

artkoshelev
artkoshelev

Reputation: 892

You can subscribe/unsubscribe when your app receives the signal from "killswitch".

Upvotes: 0

Marquis Blount
Marquis Blount

Reputation: 8095

There are a few unknowns such as are you using the java aws client? what version, is this a production environment or just for debugging/testing, is the kill switch manual or kicked off automatically.

based on a few assumptions here is how to accomplish what is asked.

if you have an sqs listener in a spring java application with the latest aws client you can do something like the following 

@SqsListener(value = mySqsQueueNameHere", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void onMessage(String payloadStr, Acknowledgment acknowledgement) throws InterruptedException, ExecutionException, JsonParseException, JsonMappingException, IOException {
    if(isKillSwitchOn) throw new SomeExceptionHere();
    .....rest of code here if kill switch not on....
}

the SqsMessageDeletionPolicy.ON_SUCCESS puts the Message back on the Queue when any exception is thrown. The SqsListener only accepts a list of Queue names and a Deletion policy I'm not seeing a clean way to shut the listener off

This would prevent the messages from being consumed and then deleted but be warned your listeners will continue to take messages off of the queue however it will put them back immediately. I'm also not sure how that effects the queues ordering not sure if SQS will allow that message to get back in line in its proper location

Upvotes: 1

Related Questions