CODEWITHSUNDEEP
djangocelerydjango-celeryamazon-sqs
tobias.mcnulty
tobias.mcnulty

Reputation: 1651

Why doesn't CeleryCAM work with Amazon SQS?

I'm using Celery 2.4.6 and django-celery 2.4.2.

When I configure Celery to use Amazon SQS per the resolution on this question: Celery with Amazon SQS

I don't see anything in the celerycam table in the Django admin. If I switch back to RabbitMQ, the tasks start showing up again.

I have a lot (now 40+) queues in SQS named something like this: "celeryev-92e068c4-9390-4c97-bc1d-13fd6e309e19", which look like they might be related (some of the older ones even have an event in them), but nothing's showing up in the database and I see no errors in the celerycam log.

Any suggestions on what the issue might be or how to debug this further would be much appreciated.

Upvotes: 30

Views: 1169

Answers (1)

enticedwanderer
enticedwanderer

Reputation: 4346

SQS is a limited implementation of an AMQP bus. As I understand, it doesn't support PUB/SUB broadcasting like say rabbit-MQ does, which is necessary for events to work properly. SNS was put in place to support broadcasting, but its a separate system.

Some libraries/packages out there are using SimpleDB as a messaging model store as a hack on top of SQS to emulate proper AMQP behavior, but apparently celery does not have a full hack in place yet.

Upvotes: 3

Related Questions