what might delay my celery tasks?

Question

lo guys,

i have a lil problem and maybe someone could help me to figure out what exactly is happening here:

i have thousands of the same tasks which should execute in milliseconds and right after i restart the worker they actually do execute in milliseconds, but right after the logging of a bunch of those tasks i get a big blob from celery saying they are done (http://d.pr/n/66h) all together in a much lager time frame .. and after that each of those tasks take around 5-13s to execute. This made me calculate that celery might be done with all of the tasks in about a week of execution time. (AAAAAhhhhh)

After a worker restart about 30-60 Task get executed in normal speed so to say. After that i get one result every few seconds - like mentioned above kinda like 4-13s. Sure in some cases there is a little bit more to do than than in others, but this does NOT justify differences from 0.08s to 13s!

Im currently using redis as the broker (in gonna switch to RabbitMQ soon) and the result backend and the task itself saves some data to a Solr instance but all of those connections should still be done in milliseconds!

Id like to be able to figure out what actually delays all those tasks. Anybody?

cheery Andy

Answer 1

after a long struggle of occasionally fixing and breaking it again i found the problem to be the worker itself (and the config - my bad). The config said to write results to the backend which it did nicely to the point where kinda 50k results where sitting in the "waste bin" .. and in time the worker became unresponsive .. after quite a few hours of fiddling around i got to the point where i was debugging the worker locally which to my surprise was stuck during the redis response method.

This kinda 'just' took around 10s on the servers but kinda like forever with my dialup connection. So i added a print to the redis response method and saw all the 50k result keys available in redis being streamed down to my machine .. for every task executed.

So this was the real reason why everything became unresponsive. Adding the settings for celery to stop wring results and clearing out the redis backend solved the problem instantly.

Id still like to know why redis likes to request keys although it should not use em. I was not evaluating the results in any way, otherwise there would not have been 50k lying around in the first place. :P

anyway - to the one guy who put that code there so long and thanx for all the fish

Answer 2

we had the same problem here. We solved the problem by playing aroung with the celery-settings in our django-settings. After decreasing concurrency (to 2) and resetting it to the original state (10) everything worked fine. Maybe give it a try.