CODEWITHSUNDEEP
djangorediscelerydjango-celery
Salvatore Iovene
Salvatore Iovene

Reputation: 2323

How can I make celery more robust with regards to dropped tasks?

Occasionally (read: too often) my celery setup will drop tasks. I'm running the latest celery 4.x on Django 1.11 with a redis backend for the queue and results.

I don't know exactly why tasks are being dropped, but what I suspect is that a worker is starting a job, then the worker is killed for some reason (autoscaling action, redeployment, out-of-memory...) and the job is killed in the middle.

At this point probably it has exited the redis queue and it won't be picked up again.

So my questions are:

  1. How can I monitor this kind of thing? I use celerymon, and the task is not reported as failed, and yet I don't see in my database the data that I expected by the task that I suspect failed.
  2. How can I make celery retry such tasks without implementing my own "fake queue" with flags in the database?
  3. How do I make celery more robust and dependable in general?

Thanks for any pointers!

Upvotes: 1

Views: 1552

Answers (1)

Diego Puente
Diego Puente

Reputation: 2004

You have to use RabbitMq instead redis, I read this in the celery documentation(right here: https://docs.celeryproject.org/en/stable/getting-started/first-steps-with-celery.html#choosing-a-broker):

RabbitMQ is feature-complete, stable, durable and easy to install. It’s an excellent choice for a production environment.

Redis is also feature-complete, but is more susceptible to data loss in the event of abrupt termination or power failures.

Using rabbit mq your problem of lossing message on restart have to gone.

Upvotes: 0

Related Questions