CODEWITHSUNDEEP
python-3.xredisrabbitmqcelery
Y A Prasad
Y A Prasad

Reputation: 391

Using Redis as Celery result backend and Message broker - Task Expiration (For key stored in redis)

I am using Celery, Redis as both the message broker and as a result backend. I am unclear about the task expiration thing or KEY expiration in Redis.

Just to be clear, when Celery client initiate the task it generates a unique task_id "celery-task-meta-64h4378-875ug43-6523g" this is what it stores in Redis as a KEY (Just example) for each task and put it in the message queue, Celery workers will then checks the message queue and execute the tasks based on the number of workers in place. If worker finished the task and mark the task as SUCCESS/FAILURE it won’t change it to PENDING or any other state.

The Celery docs say that the expiration time corresponds to time after "publishing" the task, but I couldn't find any info of what "publishing" actually means.

I know that celery stores the task as a Redis Key and has a default expiration of 1 day (86400 seconds). In my case once after the task is created and stored in Redis as a KEY it takes more time for the worker to execute the task and update the result for that task whether it may be a Success/Failure.

Question #1: Regarding Redis key expiration time..Is that 1 day default time which celery is creating counts right from the time Key is created or after the task result is updated to the key by worker (I mean key created in Redis -> worker started that task -> worker finished and updated the task (Key in redis))..?

My only concern is after celery created the new task, worker then started executing that task and takes more than one day to finish that task (worst possible case..If we have more and more number of tasks created) and in mean time if the KEY expires in Redis...Then what to do in these cases..?

Quick solution is to Increase the redis Key expiration time to more than one day :)

Question #2: Is going to RabbitMq instead of Redis in the above scenario a good choice..? In this case we can store the result in the persistent db and we don't have to worry about the expiration time and also Redis In-Memory cache fill-up cases.

Please correct me if I am worng in understanding something in the above mentioned points. Any feedback/help on this will be greatly appreciated :)

Upvotes: 3

Views: 2436

Answers (1)

2ps
2ps

Reputation: 15926

Question #1: The expire time that you have referenced in the link is from the time that the call to apply_async or delay is made.

Question #2: Either one is a good choice. Redis is slightly less reliable, but much easier to configure than rabbitmq. That said, using rabbitmq as your broker is by far the most popular choice that most devs make. YMMV.

Upvotes: 1

Related Questions