CODEWITHSUNDEEP
celerymessage-queueamazon-sqs
ChrisM
ChrisM

Reputation: 2248

Celery: am I misunderstanding how it's meant to work?

I have an EC2-based REST web service which is connected to a set of worker instances via SQS. I am hoping to replace this current setup with Celery, retaining (for the moment) SQS as the broker.

I am not sure my mental picture of how Celery operates is quite accurate, however. Following the pattern that seems to be implied in this article, my understanding was that I could enqueue messages to SQS from my API, and these would then be processed by the Celery workers in a timely fashion. The examples I have found, however (almost none of which involve using SQS as a broker), seem to indicate that Celery itself does the enqueuing. So Celery acts like a 'wrapper' for the queue, submitting and processing tasks. It's acting as both producer and consumer - is that right? If so that seems like a big problem for me, since I don't use Python. The workers themselves would use execl() to run the actual tasks, but how would I handle the initial submission of the message into the queue, from the API server? Is this something that Celery even allows?

Am I right about this, or was my original understanding accurate? Or are they equally legitimate ways in which Celery can be utilized?

Upvotes: 3

Views: 2866

Answers (2)

BitMask777
BitMask777

Reputation: 2733

Re: how to conceptualize Celery, I think of it as an asynchronous programming framework (although it is possible to use it synchronously). Identifying a function as a Celery Task decorates your function with async operations, like .delay() and .apply_async(), and returns you an AsyncResult for checking execution and status. It happens to use RabbitMQ as the default broker to manage it's internal async processing. By default it acts as both the producer and consumer of the queued tasks.

As long as you adhere to the protocol you can write your own Celery producer or consumer against RabbitMQ or the broker of your choice. A decent visual explanation of the relationship between RabbitMQ and Celery can be found here.

Upvotes: 0

mher
mher

Reputation: 10846

The protocol Celery uses for messages is well documented. You can implement a custom producer in any language or use one of the existing implementations (PHP, Node.js, Ruby, etc.). They are listed here.

Also you can use web hooks which allows to call tasks from any environment.

Upvotes: 1

Related Questions