CODEWITHSUNDEEP
djangoceleryceleryd
Quintin Par
Quintin Par

Reputation: 16252

How do I restart celery workers gracefully?

While issuing a new build to update code in workers how do I restart celery workers gracefully?

Edit: What I intend to do is to something like this.

Upvotes: 56

Views: 81580

Answers (11)

黄大胆
黄大胆

Reputation: 1

why not use python feature importlib.reload?

Upvotes: 0

ramwin
ramwin

Reputation: 6276

use TERM signal as the doc said.

kill -TERM <celerypid>

This will cause celery to Warm shutdown. It stops accepting new tasks and exit after all the current jobs finished.

Upvotes: 0

Benny Chan
Benny Chan

Reputation: 928

If you're using docker/docker-compose and putting celery into a separate container from the Django container, you can use

docker-compose kill -s HUP celery

, where celery is the container name. The worker will be gracefully restarted and the ongoing task is not brutally stopped.

Tried pkill, kill, celery multi stop, celery multi restart, docker-compose restart. All not working. Either the container is stopped abruptly or the code is not reloaded.

I just want to reload my code in the prod server manually with a 1-liner. Don't want to play with daemonization.

Upvotes: 0

armonge
armonge

Reputation: 3138

According to https://docs.celeryq.dev/en/stable/userguide/workers.html#restarting-the-worker you can restart a worker by sending a HUP signal

 ps auxww | grep celeryd | grep -v "grep" | awk '{print $2}' | xargs kill -HUP

Upvotes: 53

Denis Kanygin
Denis Kanygin

Reputation: 1133

Might be late to the party. I use:

sudo systemctl stop celery

sudo systemctl start celery

sudo systemctl status celery

Upvotes: -4

cảnh nguyễn
cảnh nguyễn

Reputation: 898

You can do:

celery multi restart w1 -A your_project -l info  # restart workers

Example

Upvotes: 9

JL Peyret
JL Peyret

Reputation: 12164

If you're going the kill route, pgrep to the rescue:

kill -9 `pgrep -f celeryd`

Mind you, this is not a long-running task and I don't care if it terminates brutally. Just reloading new code during dev. I'd go the restart service route if it was more sensitive.

Upvotes: 6

crunk_monad
crunk_monad

Reputation: 43

I have repeatedly tested the -HUP solution using an automated script, but find that about 5% of the time, the worker stops picking up new jobs after being restarted.

A more reliable solution is:

stop <celery_service>
start <celery_service>

which I have used hundreds of times now without any issues.

From within Python, you can run:

import subprocess
service_name = 'celery_service'
for command in ['stop', 'start']:
    subprocess.check_call(command + ' ' + service_name, shell=True)

Upvotes: 2

zengr
zengr

Reputation: 38899

celery multi start 1 -A proj -l info -c4 --pidfile=/var/run/celery/%n.pid
celery multi restart 1 --pidfile=/var/run/celery/%n.pid

http://docs.celeryproject.org/en/latest/userguide/workers.html#restarting-the-worker

Upvotes: 16

guettli
guettli

Reputation: 27826

What should happen to long running tasks? I like it this way: long running tasks should do their job. Don't interrupt them, only new tasks should get the new code.

But this is not possible at the moment: https://groups.google.com/d/msg/celery-users/uTalKMszT2Q/-MHleIY7WaIJ

Upvotes: 2

sebasmagri
sebasmagri

Reputation: 168

You should look at Celery's autoreloading

Upvotes: 1

Related Questions