CODEWITHSUNDEEP
pythondjangocelery
rohitnaidu19
rohitnaidu19

Reputation: 693

Celery add task dynamically to chain

I am using celery 3 with Django.
I have a list of jobs in database. User can start a particular job which starts a celery task.
Now I want user to be able to start multiple jobs and it should add them to the celery queue and process them one after the other not in parallel as with async.

I am trying to create a job scheduler with celery where user can select the jobs to execute and they will be executed in sequential fashion.
If I use chain() then I cannot add new tasks to the chain dynamically.

What is ​the best solution?

Upvotes: 9

Views: 3657

Answers (1)

zarnoevic
zarnoevic

Reputation: 449

A better primitive for you to use would be link instead of chain in Celery.

From the documentation:

s = add.s(2, 2)
s.link(mul.s(4))
s.link(log_result.s())

You can see how this allows you to dynamically add a task to be executed by iterating through the required tasks in a loop, and linking each one's signature. After the loop you would want to call something like s.apply_async(...) to execute them.

Upvotes: 2

Related Questions