CODEWITHSUNDEEP
pythoncelery
vahvero
vahvero

Reputation: 663

Limit number of active tasks in Celery

Having task

from celery import Celery
celery = Celery(__name__, **kwargs) # Some setting here probably?

@celery.task(name="task_runner", bind=True)
def generic_task(self, function_type):
    task_id = self.request.id
    # Do whatever
    if function_type == 1:
        high_ram_usage_1() # Really a high memory function
    if function_type == 2:
        high_ram_usage_2() # Another high memory function
    # Other functions

How to limit the active task count to two for the whole celery app? Only two active functions fit in my memory which eventually crashes the server on my development computer.

EDIT:

celery = Celery(__name__, concurrency=2) # Does not work
celery.conf.update(concurrency=2) # Does not work
@celery.task(concurrency=2) # Does not work
def dummy_task(self):
    time.sleep(5)

Worker logs

<redacted>:Task dummy_task[b62052bf-c893-4ad4-bc30-cffaa39bcbb1] succeeded in 5.110082127997885s: True
<redacted>:Task dummy_task[a2def96d-6306-422a-be79-14f43886aa7f] succeeded in 5.104002231993945s: True
<redacted>:Task dummy_task[4b377742-c22f-432c-a675-9c0dbdd2cb41] succeeded in 5.119215640006587s: True
<redacted>:Task dummy_task[5735de81-79f7-43ac-b28c-42cb071011ca] succeeded in 5.139429216011195s: True
<redacted>:Task dummy_task[5b37c19f-693d-45d4-8580-6b493632c5ab] succeeded in 5.142507184995338s: True
<redacted>:Task dummy_task[d8b00cfc-0773-43cd-bc40-d5a55a1dfda0] succeeded in 5.158245797007112s: True
<redacted>:Task dummy_task[0d93b652-b4f9-4bff-97b5-0660cdc05586] succeeded in 5.168131768004969s: True
<redacted>:Task dummy_task[b7946f28-7df6-42d8-9a9d-1e7b5efc3a4a] succeeded in 5.173699700011639s: True
<redacted>:Task dummy_task[4f1f9389-040a-45dd-b252-4342ba1d3445] succeeded in 5.187171609024517s: True
<redacted>:Task dummy_task[6651040f-bffa-46b6-ab4f-950d8f518b46] succeeded in 5.192372458986938s: True
<redacted>:Task dummy_task[b65d2de1-5029-47f7-8048-73c4c38065ad] succeeded in 5.279150495975045s: True
<redacted>:Task dummy_task[3211729b-e8b9-4c72-9354-caf16e7a6970] succeeded in 6.117392421991099s: True

Upvotes: 1

Views: 1344

Answers (1)

DejanLekic
DejanLekic

Reputation: 19787

You can't specify concurrency on a task, so @celery.task(concurrency=2) does not work. The thread @TharunK pointed at have already answer to your question as you have the same problem. However, it seems you did not fully understand it.

What the answer to that thread basically says is that you need to configure your Celery worker to spawn no more than N worker processes. In your case N=2.

There are few ways to do it, one of them is the --concurrency command-line option of Celery worker.

You have been quite close to set it up using the configuration object, but you used the wrong key. The correct key is worker_concurrency so celery.conf.update(worker_concurrency=2) should do the job.

Upvotes: 1

Related Questions