CODEWITHSUNDEEP
rediscelery
buxizhizhoum
buxizhizhoum

Reputation: 1929

celery with redis as broker and result backend uses too many redis connections

We have a celery cluster with redis as broker and result backend, there are about 100+ servers, and 200+ queues, currently, the redis connections(9000+) are close to the max connection threshold(10000), and it is still increasing gradually when new servers are added to the cluster.

We relay on the task state in celery to implement some more check, so we could not set ingore_result=True(This method could reduce the redis connection rapidly, but we just could not use it considering the limit of other functions), and the BROKER_POOL_LIMIT is already set to 0, but it does not have too much effect, is there any other ways to reduce the redis connection?

Upvotes: 2

Views: 2224

Answers (1)

Japneet Singh Chawla
Japneet Singh Chawla

Reputation: 206

I too have faced this issue with redis as broker and result backend, with such huge load redis not performs well due to single-threaded nature and for every task, it has to save the result as a separate key. You can try to rearchitect celery to use AMQP systems like RabbitMq for broker and redis for the result backend. Have a look at this article which explains things pretty well http://techscouter.blogspot.com/2018/05/celery-with-heavy-workloads.html

Upvotes: 4

Related Questions