CODEWITHSUNDEEP
pythonrabbitmqcelery
Yang HG
Yang HG

Reputation: 720

Celery: How to batch produce tasks?

I have a large loop to produce tasks:

for i in range(1000):
    receiver.apply_async(args=(i), kwargs={}, exchange=topic_exchange, routing_key=topic_key)

And I found there is a module celery.contrib.batches before celery 3.X or celery_batches after celery 4.X. But this module doesn't seem to support such params. So how can I do it?

I'm using celery 4.4.7 with rabbitmq.

Upvotes: 5

Views: 2755

Answers (1)

DejanLekic
DejanLekic

Reputation: 19787

If by "batch" you mean a small subset (chunk) of the all tasks, then you could look into Chunks. Instead of using chunks (they are after all made for different purpose), I suggest you use Chord if you care about results. If you do not, then just simply create a Group. Thousand tasks is nothing - we have had chords/groups made of tens of thousands of tasks and Celery copes with that load pretty well.

Upvotes: 2

Related Questions