CODEWITHSUNDEEP
pythonamazon-web-servicesceleryamazon-sqsamazon-sns
gCoh
gCoh

Reputation: 3089

Celery publish messages using SNS

I'm looking for a way publishing Celery messages to SNS topic, to be subscribed by multiple queues.

The reason I would like to do so, is that I'm trying to implement Multiple Receivers pattern using Celery

I.e every service will get all the messages were published, and execute only the relevant ones. All the rest will be acknowledged

Upvotes: 1

Views: 1387

Answers (2)

gCoh
gCoh

Reputation: 3089

The pattern we are going to implement is an event bus, where each microservice have its own queue.

I.e for each service we will have 2 queues, one subscribed to SNS topic, second for the worker. the first will be consumed by a poller, inserting new jobs to worker queue.

This will allow decoupling between two services, i.e one service can be down while the second appends new messages to the queue

Upvotes: 0

DejanLekic
DejanLekic

Reputation: 19787

You need to dig deeply into the SQS transport to find out the format of the Celery messages, and then implement publishing via SNS... Celery does not provide this out-of-box. I am sure Celery team would welcome this as some kind of utility addition, or similar. It would be a useful feature.

Upvotes: 1

Related Questions