CODEWITHSUNDEEP
djangodeploymentcelerydjango-celeryinit.d
manu
manu

Reputation: 1111

daemonizing celery process celeryd-multi not found

I'm trying to daemonize the celery process for django running inside a virtualenv. I copied the celeryd file from https://github.com/celery/celery/tree/master/extra/generic-init.d to /etc/init.d/

I then created a config file with the contents at http://ask.github.io/celery/cookbook/daemonizing.html#example-django-configuration-using-virtualenv and saved it as /etc/default/celeryd

This is my /etc/default/celeryd:

# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Where to chdir at start.
CELERYD_CHDIR="/home/manu/location/to/project/"

# Python interpreter from environment.
ENV_PYTHON="/home/manu/.virtualenvs/project_env/bin/python"

# How to call "manage.py celeryd_multi"
CELERYD_MULTI= "$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryd_multi"

# How to call "manage.py celeryctl"
CELERYCTL="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryctl"

# Extra arguments to celeryd
CELERYD_OPTS="--verbose --fake --time-limit=300 --concurrency=8"

# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="project.settings.production"

When I run:

sudo sh -x /etc/init.d/celeryd start

It fails with the following error:

celeryd-multi start w1 --uid=celery --gid=celery --workdir=/home/manu/location/to/project/ --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/%n.log --loglevel=INFO --cmd=-m celery.bin.celeryd_detach --verbose --fake --time-limit=300 --concurrency=8 /etc/init.d/celeryd: 140: /etc/init.d/celeryd: celeryd-multi: not found

I can see that it's unable to find celeryd_multi. Strangely, running

/path/to/project_env/bin/python /path/to/manage.py celeryd_multi

is showing that celeryd-multi is available.

[Update - Correcting my start command with @Daniel Roseman's input]

Now, when I run:

sh -x /etc/init.d/celeryd start

This is the output I see:

/path/to/.virtualenvs/virtenv/bin/python /path/to/project//manage.py celeryd_detach --time-limit=300 --concurrency=8 --gid=celery --broker=amqp://:@localhost:5672// -n w1.ubuntu-12-10 --logfile=/var/log/celery/w1.log --loglevel=INFO --uid=celery --pidfile=/var/run/celery/w1.pid --workdir=/path/to/project/

OK

+ sleep 5

+ exit 0

What am I missing here? Please help!

Upvotes: 3

Views: 4730

Answers (2)

sontek
sontek

Reputation: 12451

Your init.d script is referencing the system python package, not the virtualenv python. So when you are running celery inside it, you are referencing the one installed in your system site packages.

Add a reference to the virtualenv python path before calling celery and everything should work just fine.

That being said, it is usually better to use init.d to run supervisord and then have supervisord spin up all your other processes like celery and your web application.

Upvotes: 3

Daniel Roseman
Daniel Roseman

Reputation: 599926

When you execute a command using sudo sh, you're starting a completely new shell environment. Inside that environment, your virtualenv is not activated, and the virtualenv's bin directory will not be on the path, which is presumably why the executable is not found.

If you really need to run this as the superuser, you should create a wrapper script that executes the virtualenv inside the sudo call.

Upvotes: 2

Related Questions