CODEWITHSUNDEEP
pythoncelery
Derp derp
Derp derp

Reputation: 209

Does a celery chain execute tasks in a specific order?

I have a task task_main which calls other tasks. But I need them to execute in a specific order.

Celery docs say not to call them one after another with .delay() and get().

http://docs.celeryproject.org/en/latest/userguide/tasks.html#avoid-launching-synchronous-subtasks

Will using chain run them in order? I cannot find this in the docs.

@shared_task
def task_a():
    pass

@shared_task
def task_b():
    pass

@shared_task
def task_b():
    pass

@shared_task
def task_main():
    chain = task_a.s() | task_b.s() | task_c.s()
    chain()

Upvotes: 1

Views: 7020

Answers (2)

Bo Wang
Bo Wang

Reputation: 31

Maybe a more concrete example following a python data science ETL pipeline, basically, we extract data from DB, then transform the data into the expected manner, then load the data into result backend:

@app.task(base=TaskWithDBClient, ignore_result=True)
def extract_task(user_id):
    """Extract data from db w.r.t user."""
    data = # some db operations ...
    return data

@app.task()
def transform_task(data):
    """Transform input into expected form."""
    data = .... # some code
    # the data will be stored in result backend
    # because we didn't ignore result.
    return data

@app.task(ignore_result=True)
def etl(user_id):
    """Extract, transform and load."""
    ch = chain(extract_task.s(user_id),
           transform_task.s())()
    return ch

Back to your main application, you only need to call:

etl.delay(user_id)

The tasks will be executed sequentially.

Upvotes: 3

xirdneh
xirdneh

Reputation: 256

Yes, if you use chains tasks will get run one after another. Here's the correct documentation for that: http://docs.celeryproject.org/en/latest/userguide/canvas.html#chains

Upvotes: 3

Related Questions