CODEWITHSUNDEEP
celeryamazon-sqsdjango-celery
Dheeraj Pande
Dheeraj Pande

Reputation: 970

How to set different visibility_timeout for different queue in celery-sqs

I am using celery and sqs for background tasks processing. We have multiple tasks running, I want to set different visibility_timeout for different tasks; is it possible to do it in settings.py file. Current settings.py file is 

BROKER_TRANSPORT_OPTIONS = {
    'region': 'ap-southeast-1',
    'polling_interval': 10,
    'queue_name_prefix': 'dev-',
    'visibility_timeout': 43200,
}

Upvotes: 0

Views: 509

Answers (1)

sytech
sytech

Reputation: 40861

This should be possible a few different ways. I believe the most straight-forward thing you can do for your case is utilize task routing capabilities of Celery. The configuration may look something like this

app.conf.task_routes = ([
    ('feed.tasks.*', {'queue': 'feeds'}),
    ('web.tasks.*', {'queue': 'web'}),
    (re.compile(r'(video|image)\.tasks\..*'), {'queue': 'media'}),
],)

See also: message routing.

Alternatively, you can also route tasks 'on-the-fly', overriding the application defaults for the queue/routing_key on a whim when you send tasks. For example...

>>> from feeds.tasks import import_feed
>>> import_feed.apply_async(args=['http://cnn.com/rss'],
...                         queue='feed_tasks',
...                         routing_key='feed.import')

Upvotes: 0

Related Questions