CODEWITHSUNDEEP
node.jsagenda
dylanjha
dylanjha

Reputation: 2523

How to set agenda job concurrency properly

Here's an example job:

const Agenda = require('agenda')
const agenda = new Agenda({db: {address: process.env.MONGO_URL}})

agenda.define('example-job', (job) => {
  console.log('took a job -', job.attrs._id)
})

So now, let's say I queue up up 11 agenda jobs like this:

const times = require('lodash/times')
times(11, () => agenda.now('example-job'))

Now if I look in the DB I can see that there are 11 jobs queued and ready to go (like I would expect).

So now I start one worker process:

agenda.on('ready', () => {
  require('./jobs/example_job')
  agenda.start()
})

When that process starts I see 5 jobs get pulled off the queue, this makes sense because the defaultConcurrency is 5 https://github.com/agenda/agenda#defaultconcurrencynumber

So far so good, but if I start another worker process (same as the one above) I would expect 5 more jobs to be pulled off the queue so there would be a total of 10 running (5 per process), and one left on the queue.

However, when the second worker starts, it doesn't pull down any more jobs, it just idles.

I would expect that defaultConcurrency is the number of jobs that can run at any given moment per process, but it looks like it is a setting that applies to the number of jobs at any moment in aggregate, across all agenda processes.

What am I missing here or what is the correct way to specify how many jobs can run per process, without putting a limit on the number of jobs that can be run across all the processes.

Upvotes: 4

Views: 4326

Answers (1)

dylanjha
dylanjha

Reputation: 2523

The problem is that the defaultLockLimit needs to be set.

By default, lock limit is 0, which means no limit, which means one worker will lock up all the available jobs, allowing no other workers to claim them.

By setting defaultLockLimit to the same value as defaultConcurrency this ensures that a worker will only lock the jobs that it is actively processing.

See: https://github.com/agenda/agenda/issues/412#issuecomment-374430070

Upvotes: 5

Related Questions