CODEWITHSUNDEEP
pythonflaskredisrabbitmqcelery
Fernando Cordero
Fernando Cordero

Reputation: 1

Redis connections keep going up even when using pool

I have a Flask Application that uses Waitress as the server and Celery+RabbitMQ (as the broker) + Redis (as the backend) for the tasks.

I noticed that my number of redis connections keep going up even when setting up a pool for redis. The only way to clear them is to kill the application (or in my case for heroku, restart the dynos).

Here is a sample of the current code:

CLOUDAMQP_URL = os.environ.get("CLOUDAMQP_URL") or os.environ.get("RABBITMQ_URL")
HEROKU = os.environ['HEROKU']

if HEROKU == "production":
    url = urlparse(os.environ.get('REDISCLOUD_URL'))
    backend_url = f"redis://{url.username}:{url.password}@{url.hostname}:{url.port}"
else:
    backend_url = "redis://localhost:6379/0"

redis_client = redis.Redis.from_url(backend_url)
broker_url = CLOUDAMQP_URL


app = Celery('chatbot_tasks', broker=broker_url, backend=backend_url)

app.conf.task_serializer = 'json'
app.conf.result_serializer = 'json'
app.conf.broker_pool_limit = 0

What could be causing this issue?

I tried setting up a pool as such:

backend_url = os.environ['REDIS_URL']
pool = ConnectionPool.from_url(backend_url)

But no fix...

Upvotes: 0

Views: 547

Answers (2)

Lehui Chen
Lehui Chen

Reputation: 1

There is a bug for Redis.from_url() as singleton, see the issue

Upvotes: 0

Amit Nagler
Amit Nagler

Reputation: 114

As I don't know what causes the increase in connections, setting redis server timeout may be a short term solution. Once it's configured, all the client connections will be closed after it is idle for the configured time. Run once using some redis client:

CONFIG SET timeout <time in seconds>

Redis client handling

Upvotes: 0

Related Questions