CODEWITHSUNDEEP
node.jsqueuekue
Paul Grimshaw
Paul Grimshaw

Reputation: 21034

Worker processing multiple job types in Kue

I am just investigating Kue as a job processor. However I am stuck with one concept:

Say I have multiple workers, all can process 1 of 2 job types.

Code looks something like this:

queue.process("job1", 1, (job, done) => {
    console.log("processing job 1")
})
queue.process("job2", 1, (job, done) => {
    console.log("processing job 2")
})

The idea is that the service will check all queues and start the highest priority jobs. But I don't want to process more than one job type in parrallel on each worker.

The problem with this is that if a "job1" and a "job2" are added to the queue at the same time, the above process will start both in parrallel.

In reality I will have "N" job types, each with configurable parrallel runs. I don't want a separate process per job type, as this will be hard to scale. Each process should be able to handle any job type.

How can I make sure that only 1 batch of job types will be processed at a time by any one worker?

Upvotes: 0

Views: 1189

Answers (1)

Ian W
Ian W

Reputation: 71

One way around this is to only have a single queue job type, but specify a job type within the data you pass to a job and process based on that.

queue.create('alljobs', {type: 'jobType1'});
queue.create('alljobs', {type: 'jobType2'});

queue.process('alljobs', 1, (job, done) => {
  if(job.data.type === 'jobType1') {
    console.log("processing job 1");
  } else if(job.data.type === 'jobType2') {
    console.log("processing job 2");
  }
});

Upvotes: 2

Related Questions