CODEWITHSUNDEEP
pythonsqlalchemycelery
Eldloppa
Eldloppa

Reputation: 91

In what way does celery + sqlalchemy badly handle multiple workers?

According to the Celery docs, the SQLAlchemy broker has problems with "more than a few" workers and may execute tasks multiple times. I haven't been able to find more precise information on this. Does the choice of database matter? Given that the database is PostgreSQL 9.1, does someone have experience with how many "more than a few" are, and how common an occurrence this is? Does it happen all the time or only under extreme corner cases?

Upvotes: 3

Views: 529

Answers (1)

Tommaso Barbugli
Tommaso Barbugli

Reputation: 12031

Using a database as task broker is not suggested as it does not perform well compared to other broker like rabbitmq or redis (the first being the best option for celery).

This because the workers will use resources to poll the db for new tasks and to publish state and results.

Sqlalchemy should not be used as broker unless you have a very limited amount of tasks and 1 worker (aka development/staging machines)

Rabbitmq is the suggested broker for production environments and you should use it :)

Upvotes: 1

Related Questions