Reputation: 145
I use celery 3.1.23.
Task code is very simple:
@app.task(bind=True, max_retries=None, default_retry_delay=settings.CELERY_RECONNECT_TIME)
def analize_text(self, **kwargs):
print 'test'
return 1
I launch celery this command:
celery -A tasks worker --loglevel=info --concurrency=4 -n analize_text -Q analize.analize_text -Ofair
So, i use 4 CPU. The problem is that celery periodically kills workers and starts new ones instead. New worker has new number in log and new PID. Here is my log https://dpaste.de/N1Vk
Why does it do it? Is this bug?
Upvotes: 3
Views: 677
Reputation: 43265
It depends on CELERY_MAX_TASKS_PER_CHILD i.e. number of tasks a worker shall complete before being recycled, default is no limit.
http://docs.celeryproject.org/en/latest/configuration.html#std:setting-CELERYD_MAX_TASKS_PER_CHILD
So,your celery configuration might be having this limit set as some low number, hence your workers are getting recycled. You can see more information in celery stats.
Upvotes: 1