CODEWITHSUNDEEP
celeryscheduled-taskscelery-taskcelerybeat
r4v1
r4v1

Reputation: 365

What if i schedule tasks for celery to perform every minute and it is not able to complete it in time?

If I schedule the task for every minute and if it is not able to be getting completed in the time(one minute). Would the task wait in queue and it will go on like this? if this happens then after few hours it will be overloaded. Is there any solution for this kind of problems?

I am using beat and worker combination for this. It is working fine for less records to perform tasks. but for large database, I think this could cause problem.

Upvotes: 1

Views: 488

Answers (1)

ItayB
ItayB

Reputation: 11337

Task is assign to queue (RabbitMQ for example).

Workers are queue consumers, more workers (or worker with high concurrency) - more tasks could be handled in parallel.

Your periodic task produce messages of the same type (I guess) and your celery router route them to the same queue. Just set your workers to consume messages from that queue and that's all.

celery worker -A celeryapp:app -l info -Q default -c 4 -n default_worker@%h -Ofair

In the example above I used -c 4 for concurrency of four (eqv. to 4 consumers/workers). You can also start move workers and let them consume from the same queue with -Q <queue_name> (in my example it's default queue).

EDIT: When using celery (the worker code) you are initiate Celery object. In Celery constructor you are setting your broker and backend (celery used them as part of the system) for more info: http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#application

Upvotes: 1

Related Questions