CODEWITHSUNDEEP
pythondjangorediscelery
Ivansek
Ivansek

Reputation: 372

Celery restart loss scheduled tasks

I use Celery to schedule the sending of emails in the future. I put the task in celery with apply_async() and ETA setted sometimes in the future.

When I look in flower I see that all tasks scheduled for the future has status RECEIVED.

If I restart celery all tasks are gone. Why they are gone?

I use redis as a broker.

EDIT1

In documentation I found:

If a task is not acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

So you have to increase the visibility timeout to match the time of the longest ETA you are planning to use.

Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of ‘lost’ tasks in the event of a power failure or forcefully terminated workers.

Periodic tasks will not be affected by the visibility timeout, as this is a concept separate from ETA/countdown.

You can increase this timeout by configuring a transport option with the same name:

BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 43200}

The value must be an int describing the number of seconds.

But the ETA of my tasks can be measured in months or years.

EDIT 2

This is what I get when I type:

$ celery -A app inspect scheduled

{u'priority': 6, u'eta': u'2015-11-22T11:53:00-08:00', u'request': {u'args': u'(16426,)', u'time_start': None, u'name': u'core.tasks.action_du e', u'delivery_info': {u'priority': 0, u'redelivered': None, u'routing_key': u'celery', u'exchange': u'celery'}, u'hostname': u'[email protected]', u'ack nowledged': False, u'kwargs': u'{}', u'id': u'8ac59984-f8d0-47ae-ac9e-c4e3ea9c4a c6', u'worker_pid': None}}

If you look closely, task wasn't acknowledged yet, so it should stay in redis after celery restart, right?

Upvotes: 15

Views: 4072

Answers (1)

Diego Puente
Diego Puente

Reputation: 2004

you have to use RabbitMq instead redis.

RabbitMQ is feature-complete, stable, durable and easy to install. It’s an excellent choice for a production environment.

Redis is also feature-complete, but is more susceptible to data loss in the event of abrupt termination or power failures.

Using rabbit mq your problem of lossing message on restart have to gone.

Upvotes: 3

Related Questions