CODEWITHSUNDEEP
djangoiocelery
chenchiyuan
chenchiyuan

Reputation: 151

[[Django Celery]] Celery blocked doing IO tasks

I use celery to do some IO tasks, such as grab remote image, sending email to users. But celery sometimes blocked with no logs. At this time, it won't do any task i send. I have to restart it, it begin to work where it blocked.

It puzzles me for a very long time. What can i do ? And what is the best practice for distributing IO tasks with celery?

Upvotes: 2

Views: 1454

Answers (2)

Houcheng
Houcheng

Reputation: 2894

By default, celery worker fork several processes waiting for tasks request from client. For the tasks of IO pending and your system need a larger number of concurrency that handle request concurrently. Here is the command:

    celery -A tasks worker --without-heartbeat -P threads --concurrency=10

If simutanelous income requests is a lot, your concurrency level have to set higher than the size of incoming reqeust burst. The system's performance may be limited by the hardware memeory size or OS's select API. You can use celery's thread/ gevent model when concurrency is large:

    celery -A tasks worker --without-heartbeat -P threads --concurrency=1000

or

    celery -A tasks worker --without-heartbeat -P gevent --concurrency=1000

Upvotes: 1

mossplix
mossplix

Reputation: 3865

you can increase the celery concurrency

manage.py celeryd   --concurrency=3

where concurrency == number of processors

run shell command 

   grep -c processor /proc/cpuinfo

to get number of processors on your machine

Upvotes: 0

Related Questions