CODEWITHSUNDEEP
pythondjangorediscelerydjango-celery
Ross Lote
Ross Lote

Reputation: 834

Does restarting celery cause duplicate tasks?

I have an email task in celery that has an eta of 10 days from now(). However, I'm finding that some people are getting 5-6 duplicate emails at a time. I've come across this problem before with BROKER_TRANSPORT_OPTIONS set too low. Now I have this in my settings file:

BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 2592000} #30 days

So that shouldn't be a problem any more. I'm just wondering if there is anything else that can cause it. i.e. restarting celery. Celery gets restarted every time I deploy new code and that can happen 5 or more times a week so it's the only thing I can think of.

Any ideas?

Thanks.

Upvotes: 3

Views: 2460

Answers (2)

baldr
baldr

Reputation: 2999

Tasks won't restart in case of incorrect worker stop if you set CELERY_ACKS_LATE = False. In this case the task marked as acknowledged immediately after consuming. See docs.

Also make sure that your tasks have no retry enabled. If any exception happens inside task - they might retry with the same input arguments.

Another possible case - your tasks are written wrong and each run selects the same recipients set.

Upvotes: 2

tug
tug

Reputation: 211

Task duplicating is possible if worker/beat processes had not stopped correctly. How do you restart celery workers/beat? Check server for zombie celery worker and beat processes. Try to stop all celery processes, check no processes of celery exist and start it again. After all check that ps ax | grep celery shows fresh workers and only one beat.

Upvotes: 3

Related Questions