CODEWITHSUNDEEP
node.jsqueuebullbullmq
oldo.nicho
oldo.nicho

Reputation: 2289

BullMQ - add jobs to queues from different service to worker

I'm trying to figure out the best architecture for a scalable BullMQ implementation. We have a number of different services that are going to be feeding jobs into queues. In some situations we may have multiple different services feeding jobs into the same queue.

Initially I had thought to contain all BullMQ implementation on a single instance and stand up a simple API with an endpoint that can receive jobs to be added to the queue. So for any service that wants to add a job to a queue, they just hit a specific endpoint and the job gets added to the queue.

I was wondering though whether an alternative approach could be to instantiate a BullMQ queue on the various services that want to add jobs to queues, and then just have the workers located on a separate service to pick up jobs from the queue when they are ready for execution? This 'worker box' can then horizontally scale up as required.

If this approach is possible, I have concerns about what the implications may be of having multiple services adding jobs to the same queue - can this cause issues or is BullMQ designed to handle such a situation?

I'm finding it difficult to find information about what standard 'best-practice' approaches are for BullMQ implementation. Any guidance greatly appreciated. Thanks.

Upvotes: 2

Views: 2316

Answers (1)

Pavel Bely
Pavel Bely

Reputation: 2415

Multiple services adding jobs to the same queue approach is fine.
Here's how you can do that using BullMQ API:

const queue = new Queue('Cars');
await queue.add('paint', { color: 'red' });

Mind that queue here is an instance of facade class that serves as an api to add jobs to queues in Redis, not the queue itself.
So it's ok for each service to have their own instance of such facade, as the source of truth is Redis.

Upvotes: 4

Related Questions