CODEWITHSUNDEEP
djangoceleryvirtualenvproduction-environment
Krish V
Krish V

Reputation: 546

Django celery WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) error

What is the best practice to run celery as a daemon in a production virtualenv? I use the following in the local environment which works perfect and receiving tasks works as expected. But in production always stuck at 

WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) error

I use the following configuration in local and in production:

/etc/default/celeryd:

CELERY_BIN="path/to/celery/bin"
CELERY_APP="myproj"
CELERYD_CHDIR="home/myuser/project/myproj"
CELERYD_OPTS="--time-limit=300 --concurrency=4"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_USER="myuser"
CELERYD_GROUP="myuser"
CELERY_CREATE_DIRS=1

/etc/init.d/celeryd:[celeryd]

Package & OS version info:

  1. Ubuntu == 16.04.2
  2. Celery == 4.1.0
  3. rabbitmq == 3.5.7
  4. django == 2.0.1

I also use these commands while making celery to run as daemon:

  1. sudo chown -R root:root /var/log/celery/
  2. sudo chown -R root:root /var/run/celery/
  3. sudo update-rc.d celeryd defaults
  4. sudo update-rc.d celeryd enable
  5. sudo /etc/init.d/celeryd start

Here is my django settings.py configuration for celery:

CELERY_BROKER_URL = 'amqp://localhost'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_RESULT_BACKEND = 'db+sqlite:///results.sqlite'
CELERY_TASK_SERIALIZER = 'json'

Need expert advise to make the celery daemon to work correctly in production virtualenv. Thanks in advance!

Upvotes: 3

Views: 8019

Answers (4)

Manish Paul
Manish Paul

Reputation: 183

In case of docker, I just allocated more resources and it worked.

enter image description here

Upvotes: 0

Krish V
Krish V

Reputation: 546

I got this error due to out of memory error caught in

/var/log/kern.log

I have Tensorflow running in one of my tasks, which needs additional computational power, but my physical memory(RAM) is not sufficient to handle that much load. I weird that there's no log in celery except SigKill 9 error. But the kernel log helped me to fix it.

Upvotes: 2

mrichter
mrichter

Reputation: 103

I guess this can be a symptom of OOM. I was deploying a Celery backend in a Docker container - "it worked on my machine" but not in the cluster. I allocated more ram to the task and no longer have the problem.

Upvotes: 1

justcompile
justcompile

Reputation: 3542

Unless you've created a seperate vhost & user for rabbitmq, set the CELERY_BROKER_URL to amqp://guest@localhost//

Also, rather than root, you should set the owner of /var/log/celery/ and /var/run/celery/ to "myuser" as you have set in your celeryd config

Upvotes: 1

Related Questions