CODEWITHSUNDEEP
pythoncelerydjango-celerygevent
Octodone
Octodone

Reputation: 585

How can I control memory with Celery running on gevent?

We run Celery with gevent, and though memory consumption starts out great, over time, the memory usage grows. Assuming we're not able to track down why the memory bloats or leaks, is there a good way to control memory usage?

When running on prefork, we had used --max-tasks-per-child and --max-memory-per-child, but these don't seem to be compatible with gevent (Celery documentation says it's only available for prefork: https://docs.celeryproject.org/en/stable/userguide/workers.html#max-tasks-per-child-setting).

Could we recreate that functionality, so that the Celery process restarts itself if memory gets too high?

Upvotes: 5

Views: 443

Answers (1)

Tristan Perotti
Tristan Perotti

Reputation: 1

You could build a cron job to restart the celery workers at a frequency of your choosing. Make sure you initiate a warm shutdown or else you will lose any tasks that are currently running. You may also consider turning down the concurrency or else reevaluate whether you should be using preforked after all.

Upvotes: -1

Related Questions