CODEWITHSUNDEEP
publish-subscribeamazon-sqs
viren
viren

Reputation: 1103

Using Single SQS for multiple subscribers based on message identifier

We have application where multiple subscribers are writing to publisher Kafka topic This data is then propagated to specific subscriber topic then subscriber consumes this data from specific topic assigned to it.

We want to use SQS for same purpose but issue is we will again need an SQS for each subscriber. Handling these multiple SQS will create an issue and if there is time when no data is published to subscriber the queue assigned to it will be idle.

Is there any way i can use single SQS from which all subscribers can consumed messages base don message identifier. Challenges needs to be cover in this design are:

  1. Each subscriber can get its message based on identifier
  2. Latency must not be there in case one publisher publish very few messages and other one is publishing it in millions.
  3. We can have one SQS for each publisher but single SQS for all subscribers of this publisher.

Can any one suggest any architecture using similar implementation.

Thanks

Upvotes: 1

Views: 1060

Answers (1)

A.Khan
A.Khan

Reputation: 3992

I think you can achieve it by setting up a single SQS queue. You would want to set up a Lambda trigger on that queue which will serve as a Service Manager (SM). SM will have a static JSON file that define the mapping between message identifier and its subscriber/worker. SM will receive an SQS message event, find the message attribute used for identifier, and then look up in JSON to find the corresponding subscriber. If subscriber is found, SM will invoke it.

Consider using SQS batch trigger.

enter image description here

Upvotes: 1

Related Questions