CODEWITHSUNDEEP
pythonrabbitmqcelery
Kasper van Rotterdam
Kasper van Rotterdam

Reputation: 50

distributing celery workers by consumer


here is a little background for my question.
We have a converter that converts xml to api calls.
we have a lot of customers converting xml's and now some customers have to wait until a customer finished converting 100's of xml files.
we have celery working with rabbitmq

what i'm trying to achieve is:
assign all celery workers to one customer until another customer sends xml's if there is another customer in the queu assign half the celery workers to another customer.

Can someone point me in the right direction, i can't seem to figure out where to start looking. Sorry for my weird sentences english is not my first language.

Upvotes: 2

Views: 76

Answers (1)

Artem Rys
Artem Rys

Reputation: 147

About what your are trying to achieve - too complicated for Celery, I think.

The obvious answer is to increase number of Celery workers using concurrency option while starting worker. It is easy to do and you do not need to change your code. I would recommend it.

Another option that is not so easy, however, can be a beautiful solution, is to use separate RabbiMQ virtual host for each customer. I can’t tell you how to do it completely in this case, but it could your starting point.

Upvotes: 1

Related Questions