CODEWITHSUNDEEP
celerycelery-taskrate-limiting
Tianbing Leng
Tianbing Leng

Reputation: 600

Django Celery Rate Limit setting doesn't work as expected

I use below command to create one worker:

celery -A proj worker -l info --concurrency=50 -Q celery,token_1 -n token_1

And in my task, I set the rate limit to 4000/m.

However, when I start running the collection, I noticed the average task processed is just around 10-20/s (with rate limit rule 4000/m enabled). Then, I removed the rate limit rule, now the task rates goes to around 60/s.

I am confused, since my rate limit is 4000/m, which is relatively 65/s. Why it finally goes just 10-20/s????? (I have already set 50 threads for the worker....)

Upvotes: 1

Views: 680

Answers (1)

2ps
2ps

Reputation: 15946

You're misunderstanding how the rate limits operate in celery. 'According to the documentation on version 4.2:

The rate limits can be specified in seconds, minutes or hours by appending “/s”`, “/m” or “/h” to the value. Tasks will be evenly distributed over the specified time frame.

Example: “100/m” (hundred tasks a minute). This will enforce a minimum delay of 600ms between starting two tasks on the same worker instance.

In essence, celery was adding a forced delay between your tasks. Since each task was already processing in about 16ms (1/60 of a second), adding another 16 ms forced delay between tasks reduced the rate at which they would process.

Upvotes: 2

Related Questions