CODEWITHSUNDEEP
pythondjangocelery
nymarya
nymarya

Reputation: 21

How to execute independent tasks sequentially using Celery?

I have to schedule some tasks which seem very complex to run in parallel. They do not depend on the result of each other and the function expects 3 arguments.

I already tried using chain, map and starmap methods. With chain I get this error:

[2019-04-23 15:28:00,991: ERROR/PoolWorker-3] Task proj.apps.tasks.generate[112a7426-5ac3-4cd6-8416-5591c3c018a3] raised unexpected: TypeError('get expected at least 1 arguments, got 0',)
Traceback (most recent call last):
  File ".../local/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File ".../local/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File ".../tasks.py", line 966, in generate
    return res.get()
TypeError: get expected at least 1 arguments, got 0

Using map I cannot pass all the arguments and with starmap all the tasks are started simultaneously. 

[2019-04-23 15:48:00,991: INFO/MainProcess] Received task: generate[..]
[2019-04-23 15:48:00,991: INFO/MainProcess] Received task: generate[..]
[2019-04-23 15:48:00,991: INFO/MainProcess] Received task: generate[..]

An example of the task:

@shared_task
def generate(field1, field2, field3=None):
   if field3 is not None:
      return field1 + field2 + field3
   return field1 + field2

Code using chain:

res = chain(generate.s(i, 5, j) for i in array1 for j in array2)
return res.get()

Code using starmap:

arguments = [(i, 4, j) for i in array1 for j in array2]
~generate.starmap(arguments)

Upvotes: 1

Views: 2633

Answers (2)

nymarya
nymarya

Reputation: 21

All I needed to do was to make a chain, as below:

res = chain(generate(i, 2, j)for i in array1 for j in array2)()
return res.get()

and then run celery with a additional argument that sets the maximum number of threads

celery -A tasks worker --concurrency=1

Upvotes: 1

2ps
2ps

Reputation: 15926

if the tasks are truly independent, you should be using .si and not .s:

tasks = chain(generate.si(i, 5, j) for i in array1 for j in array2)
res = tasks()
return res.get()

Upvotes: 1

Related Questions