CODEWITHSUNDEEP
djangopython-2.7celerydjango-celeryceleryd
Karthikeyan Selvaraj
Karthikeyan Selvaraj

Reputation: 1

Celery does not start multiple workers

I am trying to set up a server to run multiple workers on Ubuntu using celery. Set up the daemon using the generic scripts and using rabbit-mq as the broker.

celery==3.1.23
django-celery==3.1.17
django-celery-beat==1.0.1

/etc/default/celeryd - (internally using celery multi start RRR SSS TTT STST OTS ...)

CELERYD_NODES="RRR SSS TTT STST OTS"
CELERYD_OPTS="-c 4 -Q:RRR r,e,h -Q:SSS s,p -Q:TTT d -Q:STST sd -Ofair --detach --time-limit=1500
CELERYD="/x/home/ks/wb/manage.py celeryd"

$ service celerd start

$ ps -ef | grep celery

/etc/init.d/celeryd start
root     25636 25631  0 01:37 pts/4    00:00:00 /usr/bin/python /usr/local/bin/celery multi start RRR SSS TTT STST OTS -c 10 --uid=celery --gid=celery --workdir=/x/home/ks/wb --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/%n.log --loglevel=DEBUG --cmd=/x/home/ks/wb/manage.py celeryd -Q:RRR r,e,h -Q:SSS s,p -Q:TTT d -Q:STST -Ofair --detach --time-limit=1500
celery   27440 25636  0 01:53 pts/4    00:00:01 [celeryd: STD@dt:MainProcess] -active- (--time-limit=1500 -c 4 --executable=/usr/bin/python --gid=celery --detach --logfile=/var/log/celer/STD.log -n STD@dt-ss-app-3040 --loglevel=DEBUG --uid=celery --pidfile=/var/run/celery/STD.pid --workdir=/x/home/ks/wb -Ofair -Q standard)                          
celery   27452 27440  1 01:53 pts/4    00:00:05 [celeryd: STD@dt:Worker-1]                                                                                                                                                                                                                                                                                                                     
celery   27453 27440  0 01:53 pts/4    00:00:01 [celeryd: STD@dt:Worker-2]                                                                                                                                                                                                                                                                                                                     
celery   27455 27440  0 01:53 pts/4    00:00:01 [celeryd: STD@dt:Worker-3]

Only one worker (STD) gets started. The other workers are not starting.

when I tried to stop the service

service celeryd stop

Worker STD stops and another worker RRR starts. which signifies that my server is able to run only one worker at a given point. Only after running celeryd stop the 5 times(number of workers) the service comes down

is there a specific reason for this behavior ? do I have to set any variable to get all the 5 nodes working?

I want to get all the nodes up and running on the same server.

Upvotes: 0

Views: 1002

Answers (1)

unixia
unixia

Reputation: 4320

What you're using is not the preferred way of using celery with django now. Please consider following the steps here. Also, you should try looking up the number of workers with

ps auxww | grep 'celery worker' | awk '{print $2}'

Upvotes: 0

Related Questions