CODEWITHSUNDEEP
pythoncelerygeventcelery-task
Vimal
Vimal

Reputation: 339

Celery + Gevent pool hangs after 1000+ tasks execution

We have 8 Core, 16 GB memory, Linux server running celery, It is running a celery worker queue myQueue, and running with 1000 concurrency under gevent pool.

After executing tasks for around 1 hour, worker suddenly freezes, it is not accepting new tasks from celery beat here is our configuration for celery

App =  Celery('tasks')
class Conf:
   BROKER_URL   = 'amqp://<user>:<pass>@<host>:<port>/<vhost>'
   CELERY_IGNORE_RESULT = True
   CELERY_IMPORTS = ("worker_class",)
   CELERYBEAT_SCHEDULE = {
       'RunTask':{
           'task': 'tasks.worker.MyWorker',
           'schedule' : timedelta(minutes=5)
       }
   }

App.config_from_object(Conf)

we are running celery like below

celery worker --workdir=tasks/ -A worker -P gevent -c 1000 -Q myQueue --loglevel=INFO

And also can someone explain how can I use gevent pool using celery multi

Upvotes: 7

Views: 3901

Answers (1)

Amir Rustamzadeh
Amir Rustamzadeh

Reputation: 4528

To specify a pool type with celery multi:

celery -A myApp multi start 4 -l INFO -P gevent -c 1000 -Q myQueue

The above command starts 4 gevent workers with each working having a concurrency level of 1000, and all consuming from myQueue.

But that's not where the fun ends, because you can even specify the concurrency of each worker, and also specify which queue each worker consumes. For example:

celery -A myApp multi start 4 -l INFO -P gevent -c:1-3 1000 -c:4 200 -Q:1-2 myQueue1 -Q:3 myQueue2 -Q:4 myQueue3

Like before we start 4 gevent workers, but now workers 1 to 3 have a concurrency of 1000, while the last worker has a concurrency of 200. Also, workers 1 and 2 consume from myQueue1, worker 3 consumes from myQueue2, and worker 4 consumes from myQueue4.

Note: The celery worker options work with celery multi.

Upvotes: 6

Related Questions