CODEWITHSUNDEEP
pythoncelery
ensignavenger
ensignavenger

Reputation: 55

Celery task's finally clause doesn't execute, how do I cleanup?

I have a celery task with a finally clause in it. I have found that some times that clause does not execute. I think this happens when the task times out, and it looks like celery kills it.

What is the best practice for having code that gets executed after a celery task- no matter what?

This wouldn't work on task success, but I can use the existing finally statement for that, and just call the cleanup code in on_failure if it would work for timeouts as well. The cleanup code is idempotent.

I considered using a chain to have a new task executed that would perform cleanup- but the second task would only execute if the first one succeeds.

I looked through the docs and found an on_failure task method (see https://gist.github.com/darklow/c70a8d1147f05be877c3) I could over ride- but I could not be sure if that executes on time outs?

In short: I need a way to have code that executes after the task completes- whether successful, failure, or it times-out.

Upvotes: 1

Views: 689

Answers (1)

2ps
2ps

Reputation: 15956

Well if your worried about timeouts why not catch the SoftTimeLimitExceeded

from celery.exceptions import SoftTimeLimitExceeded

@app.task(name='my_task', soft_time_limit=1000, time_limit=2000)
def my_task():
    try:
        do_stuff()
    except SoftTimeLimitExceeded:
        raise
    finally:
        do_some_cleanup.delay()

Upvotes: 1

Related Questions