CODEWITHSUNDEEP
pythoncelerycelery-task
nolwww
nolwww

Reputation: 1715

Assign a specific number of workers for each Celery task in Python

I have three Celery tasks:

And I have 4 cpus in the machine. I would like to allocate 3 workers for prediction and training, and 1 for healthcheck. What is the easiest way to implement that? Note that I am going to schedule these tasks, and thus, I can't specify the number of workers in the apply_async() function directly.

Actual configs:

CELERY = Celery(
    CELERY_APP_NAME,
    backend=CELERY_BACKEND,
    broker=CELERY_BROKER,
    include=["src.tasks"],
)

CELERY.conf.update({"task_routes": {"src.tasks.*": {"queue": "input_queue"}},
                    }
                   )


@CELERY.task
def prediction():
    pass

@CELERY.task
def training():
    pass

@CELERY.task
def healthcheck():
    pass

And the command to run the worker:

celery --loglevel=INFO -A src.tasks worker -Q input_queue

Upvotes: 1

Views: 1010

Answers (1)

DejanLekic
DejanLekic

Reputation: 19787

This is how I would do it, because I could never truly understand celery multi:

  • celery -A yourproject.yourapp -l info -c 3 -Q prediction
  • celery -A yourproject.yourapp -l info -c 1 -Q healthcheck

After you run something similar to the above (with correct application parameter) you will end up with two workers subscribed to different queues. Your prediction and training tasks would be triggered with queue=prediction among other named parameters, while healthcheck task should be sent to the healthcheck queue in the similar way.

Upvotes: 1

Related Questions