CODEWITHSUNDEEP
pythoncelery
Leon
Leon

Reputation: 6554

Celery - one task in one second

I use Celery to make requests to the server (in tasks). I have hard limit - only 1 request in one second (from one ip). I read this, so its what I want - 1/s. In celeryconfig.py I have:

CELERY_DISABLE_RATE_LIMITS = False

CELERY_DEFAULT_RATE_LIMIT = "1/s"

But I have the messages, that I have too many requests per second.

In call.py I use groups.

I think, rate_limits does not work, because I have a mistake in celeryconfig.py.

How to fix that? Thanks!

Upvotes: 0

Views: 1003

Answers (1)

Chillar Anand
Chillar Anand

Reputation: 29594

When you start a celery worker with 

celery -A your_app worker -l info

the default concurrency is equal to the number of the cores your machine has. So,eventhough you set a rate limit of '1/s', it is trying to process multiple tasks concurrently.

Also setting a rate_limit in celery_config is a bad idea. Now you have only one task, if you add new tasks to your app, the rate limits will affect each other.

A simple way to achieve your one task per one second is this.

tasks.py

import time

from  celery  import  Celery
app = Celery('tasks', backend='amqp', broker='amqp://guest@localhost//')

@app.task()
def task1():
    time.sleep(1)
    return('task1')

Now start you worker with a concurrency of ONE

celery -A my_taks.py worker -l info -c 1

This will execute only one task per second. Here is my log with the above code.

[2014-10-13 19:27:41,158: INFO/MainProcess] Received task: task1[209008d6-bb9d-4ce0-80d4-9b6c068b770e]
[2014-10-13 19:27:41,161: INFO/MainProcess] Received task: task1[83dc18e0-22ec-4b2d-940a-8b62006e31cd]
[2014-10-13 19:27:41,168: INFO/MainProcess] Received task: task1[e1b25558-0bb2-405a-8009-a7b58bbfa4e1]
[2014-10-13 19:27:41,171: INFO/MainProcess] Received task: task1[2d864be0-c969-4c52-8a57-31dbd11eb2d8]
[2014-10-13 19:27:42,335: INFO/MainProcess] Task task1[209008d6-bb9d-4ce0-80d4-9b6c068b770e] succeeded in 1.170940883s: 'task1'
[2014-10-13 19:27:43,457: INFO/MainProcess] Task task1[83dc18e0-22ec-4b2d-940a-8b62006e31cd] succeeded in 1.119711205s: 'task1'
[2014-10-13 19:27:44,605: INFO/MainProcess] Task task1[e1b25558-0bb2-405a-8009-a7b58bbfa4e1] succeeded in 1.1454614s: 'task1'
[2014-10-13 19:27:45,726: INFO/MainProcess] Task task1[2d864be0-c969-4c52-8a57-31dbd11eb2d8] succeeded in 1.119111023s: 'task1'

Upvotes: 2

Related Questions