CODEWITHSUNDEEP
pythoncelerygevent
pomo_mondreganto
pomo_mondreganto

Reputation: 2074

Celery's @shared_task not working with soft_time_limit

Here's some simple code:

@shared_task(time_limit=10, soft_time_limit=5)
def check_action():
    try:
        __import__("time").sleep(100)
    except celery.exceptions.SoftTimeLimitExceeded as e:
        print("Here", e, type(e))
    except BaseException as e:
        print("There", e, type(e))

It's a shared_task with time_limit and soft_time_limit set up, so I expect Here to be printed after 5 seconds (along with SoftTimeLimitExceeded exception info) and then There to be printed with some hard-timeout exception.

Instead, it doesn't raise SoftTimeLimitExceeded at all, only hard time limit is called, full output is: 

celery_1               | [2019-05-31 21:36:36,709: WARNING/MainProcess] There
celery_1               | [2019-05-31 21:36:36,710: WARNING/MainProcess] 10 seconds
celery_1               | [2019-05-31 21:36:36,710: WARNING/MainProcess] <class 'gevent.timeout.Timeout'>

As it can be seen from logs, I use gevent workers, if it's of any help.

UPD #1: the problems seems to be with gevent pool (is I remove --pool=gevent, everything works perfectly).

UPD #2: eventlet pool doesn't seem to support any timeout, even the hard one.

Upvotes: 1

Views: 2673

Answers (1)

pomo_mondreganto
pomo_mondreganto

Reputation: 2074

It really is the problem with gevent pool, as mentioned in this pull request.

Question can be considered closed.

Upvotes: 2

Related Questions