CODEWITHSUNDEEP
pythondjangomultithreadingconcurrencycelery
MiniYuuzhan
MiniYuuzhan

Reputation: 118

CELERYD_CONCURRENCY, --concurrency and autoscale

I have a few questions regarding task routing, concurrency and performance. Here is my use case :

I've got one dedicated server to run celery tasks, so I can use all the CPUs to run celery workers on this server.

I have a lot of different python tasks, which I route using : CELERY_ROUTES and because the tasks perform really different types of python code, I created 5 different workers. These worker are created when I deploy my project using ansible, here is an example:

[program:default_queue-celery]
command={{ venv_dir }}/bin/celery worker --app=django_coreapp --loglevel=INFO --concurrency=1 --autoscale=15,10 --queues=default_queue
environment =
    SERVER_TYPE="{{ SERVER_TYPE }}",
    DB_SCHEMA="{{ DB_SCHEMA }}",
    DB_USER="{{ DB_USER }}",
    DB_PASS="{{ DB_PASS }}",
    DB_HOST="{{ DB_HOST }}"
directory={{ git_dir }}
user={{ user }}
group={{ group }}
stdout_logfile={{ log_dir }}/default_queue.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=5
redirect_stderr=true
autostart=true
autorestart=true
startsecs=10
killasgroup=true

I have also a CELERY_QUEUES in settings.py to make the bridge between CELERY_ROUTES and my celery programs (Queues)

CELERY_DEFAULT_QUEUE = 'default_queue'

And if it happens that I don't route a task it will go to my 'default_queue'

To give space to all of my queues, I set --concurrency to 1 for default_queue, and more for my most important queue.

But I am wondering, does AutoScale have impact on the same value as concurrency ? Meaning, if I set concurrency to 1 and --autoscale to 15,10 (example above)

Will my worker 'work' on CPU and process a maximum of 15 tasks on this CPU ? Or does this mean something completely different ?

Upvotes: 4

Views: 6232

Answers (1)

bruno desthuilliers
bruno desthuilliers

Reputation: 77942

It makes no sense setting both concurrency and autoscale since both are means to control the number of worker subprocesses for a given worker instance, as explained here.

--concurrency N means you will have exactly N worker subprocesses for your worker instance (meaning the worker instance can handle N conccurent tasks).

--autoscale max, min means you will have at least min and at most max concurrent worker subprocesses for a given worker instance.

On which CPU each process (the main worker process or any of it's child subprocesses) will run is not predictable, it's an OS thing, but do not assume subprocesses will all run on the same CPU (chances are they won't - that's part of the point of having concurrent subprocesses actually).

Upvotes: 9

Related Questions