CODEWITHSUNDEEP
djangopostgresqlcelerydjango-celeryeventlet
Leb
Leb

Reputation: 117

Celery eventlet worker threads using too many database connections

I have 2 celery workers which pool via eventlet, config is below:

celery multi start w1 w2 -A proj -l info --time-limit=600 -P eventlet -c 1000

When running more than 100 tasks at a time, I get hit by the error:

OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections

I'm running on PostgreSQL with max. connections set at the default of 100.

From what I read online, I thought worker threads in the pools would share the same DB connection. However, mine seem to try and create one connection per thread, which is why the error occurs.

Any ideas?

Thanks!

Upvotes: 3

Views: 2423

Answers (1)

temoto
temoto

Reputation: 5577

Django has (or had?) idle DB connection reuse to avoid overhead of creating new connection for each request. Idle reuse is not relevant in this scenario.

Django never had limiting DB connection pool. (please correct if wrong)

Consider overall design:

  • how many tasks do you need to execute concurrently? (real numbers are often not nice powers of 10)
  • how many simultaneous connections from this application can your database sustain?
  • do you need to place artificial bottlenecks (pools) or do you need to increase limits and use available hardware?

Consider using external [Postgresql connection pool] (google terms in square braces) or include one somewhere in your application.

Upvotes: 1

Related Questions