CODEWITHSUNDEEP
pythonceleryvirtualenvmessage-queue
okoboko
okoboko

Reputation: 4482

Celery: Start Worker Automatically (on boot)

I have tasks (for Celery) defined in /var/tasks/tasks.py.

I have a virtualenv at /var/tasks/venv which should be used to run /var/tasks/tasks.py.

I can manually start a worker to process tasks like this:

cd /var/tasks
. venv/bin/activate
celery worker -A tasks -Q queue_1

Now, I want to daemonize this.

I copied the init.d script from GitHub and am using the following config file in /etc/default/celeryd:

# name(s) of nodes to start
CELERYD_NODES="worker1"

# absolute or relative path to celery binary
CELERY_BIN="/var/tasks/venv/bin/celery"

# app instance
CELERY_APP="tasks"

# change to directory on upstart
CELERYD_CHDIR="/var/tasks"

# options
CELERYD_OPTS="-Q queue_1 --concurrency=8"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# unprivileged user/group
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# create pid and log directories, if missing
CELERY_CREATE_DIRS=1

When I start the service (via the init.d script), it says:

celery init v10.1.
Using config script: /etc/default/celeryd

But, it does not process any tasks from the queue, nor is there anything in the log file.

What am I doing wrong?

Upvotes: 2

Views: 6174

Answers (3)

bilbohhh
bilbohhh

Reputation: 823

In case you use systemd, you should enable a celery service. It will activate your celery daemon on boot.

sudo systemctl enable yourcelery.service

Upvotes: 1

Shivam Kotwalia
Shivam Kotwalia

Reputation: 1503

Supervisor might be a good option but if you want to use Celery Init.d Script will recommend you to copy it from their Github Source. 

sudo vim /etc/init.d/celeryd

Copy the code from https://github.com/celery/celery/blob/master/extra/generic-init.d/celeryd in to the file. See daemonizing tutorial for details.

sudo chmod 755 /etc/init.d/celeryd
sudo chown root:root /etc/init.d/celeryd
sudo nano /etc/default/celeryd

Copy paste the below config and change accordingly

#Where your Celery is present
CELERY_BIN="/home/shivam/Desktop/deploy/bin/celery"

# App instance to use 
CELERY_APP="app.celery"

# Where to chdir at start
CELERYD_CHDIR="/home/shivam/Desktop/Project/demo/"

# Extra command-line arguments to the worker 
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# %n will be replaced with the first part of the nodename. 
CELERYD_LOG_FILE="/var/log/celery/%n%I.log" 
CELERYD_PID_FILE="/var/run/celery/%n.pid"


# Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# A user/group combination that already exists (e.g., nobody). 
CELERYD_USER="shivam" 
CELERYD_GROUP="shivam"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured. 
CELERY_CREATE_DIRS=1
export SECRET_KEY="foobar"

Save and exit

sudo /etc/init.d/celeryd start
sudo /etc/init.d/celeryd status

This will auto start Celery on Boot

sudo update-rc.d celeryd defaults

Upvotes: 4

okoboko
okoboko

Reputation: 4482

I ended up using Supervisor and a script at /etc/supervisor/conf.d/celery.conf similar to this:

https://github.com/celery/celery/blob/3.1/extra/supervisord/celeryd.conf

This handles demonization, among other things, quite well and automatically.

Upvotes: 0

Related Questions