CODEWITHSUNDEEP
amazon-web-servicesparallel-processingamazon-sqsamazon-sns
kishore k
kishore k

Reputation: 147

Publishing messages from AWS SNS to SQS

As there is a limitation on SQS to support multiple consumers to process messages in parallel. ie. m1 to m10 picked by process 1 and m11 to m20 picked by process 2 and so on.. with duplication. Since this is not supported by SQS, I am thinking of using SNS + SQS (list of queues subscribed), where each process listens to its specific queue and processes records.

Is there an option to set between SNS and SQS like round-robin so that SNS distributes messages to SQS in a round robin fashion, So that each queue would have unique messages without duplication across queues?

Thanks in advance!

Regards, Kumar

Upvotes: 0

Views: 1405

Answers (2)

Arafat Nalkhande
Arafat Nalkhande

Reputation: 11718

Unless you are using SQS FIFO queues, your assumption about the limitation of SQS not supporting multiple parallel consumers is not correct

Standard SQS do support multiple parallel consumers.

Regarding the SQS FIFO queues they don't serve messages from the same message group to more than one consumer at a time. However, if your FIFO queue has multiple message groups, you can take advantage of parallel consumers, allowing Amazon SQS to serve messages from different message groups to different consumers.

Upvotes: 1

Matt Healy
Matt Healy

Reputation: 18531

If you don't want your SNS publish to go to all subscribers (queues), look in to SNS Message Filtering. Message filtering allows you to define logic controlling which subscribers receive a given message.

By default, a subscriber of an Amazon SNS topic receives every message published to the topic. To receive only a subset of the messages, a subscriber assigns a filter policy to the topic subscription.

A filter policy is a simple JSON object. The policy contains attributes that define which messages the subscriber receives. When you publish a message to a topic, Amazon SNS compares the message attributes to the attributes in the filter policy for each of the topic's subscriptions. If there is a match between the attributes, Amazon SNS sends the message to the subscriber. Otherwise, Amazon SNS skips the subscriber without sending the message to it. If a subscription lacks a filter policy, the subscription receives every message published to its topic.

Upvotes: 1

Related Questions