CODEWITHSUNDEEP
multiprocessingceleryworkercelery-taskbroker
jeldikk
jeldikk

Reputation: 353

How does Celery work?

I have recently started working on distributed computing for increasing the computation speed. I opted for Celery. However, I am not very familiar with some terms. So, I have several related questions.

From the Celery docs:

What's a Task Queue?

...

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, the broker then delivers that message to a worker.

What are clients (here)? What is a broker? Why are messages delivered through a broker? Why would Celery use a backend and queues for interprocess communication?

When I execute the Celery console by issuing the command

celery worker -A tasks --loglevel=info --concurrency 5

Does this mean that the Celery console is a worker process which is in charge of 5 different processes and keeps track of the task queue? When a new task is pushed into the task queue, does this worker assign the task/job to any of the 5 processes?

Upvotes: 16

Views: 11664

Answers (2)

Ahtisham
Ahtisham

Reputation: 10126

We can take analogy of assembly line packaging in a factory to understand the working of celery.

enter image description here

  1. Each product is placed on a conveyor belt.
  2. The products are processed by machines.
  3. At the end all the processed product is stored in one place one by one.

Celery working:

Note: Instead of taking each product for processing as they are placed on convey belt, In celery the queue is maintained whose output will be fed to a worker for execution one per task (sometimes more than one queue is maintained).

  1. Each request (which is a task) is send to a queue (Redis/Rabbit MQ) and an acknowledgment is send back.
  2. Each task is assigned to a specific worker which executes the task.
  3. Once the worker has finished the task its output is stored in the result backend (Redis).

Upvotes: 0

scytale
scytale

Reputation: 12641

Last question first:

celery worker -A tasks --loglevel=info --concurrency 5

You are correct - the worker controls 5 processes. The worker distributes tasks among the 5 processes.

A "client" is any code that runs celery tasks asynchronously.

There are 2 different types of communication - when you run apply_async you send a task request to a broker (most commonly rabbitmq) - this is basically a set of message queues.

When the workers finish they put their results into the result backend.

The broker and results backend are quite separate and require different kinds of software to function optimally.

You can use RabbitMQ for both, but once you reach a certain rate of messages it will not work properly. The most common combination is RabbitMQ for broker and Redis for results.

Upvotes: 12

Related Questions