CODEWITHSUNDEEP
rubyruby-on-rails-3ruby-on-rails-3.1delayed-job
slotishtype
slotishtype

Reputation: 2751

Delayed_job - Multiple parallel queues?

I am using delayed_job and moved to a new beefier server. So now I would like to run parallel jobs, as now I have the POWER!, but am confused on whether delayed_job can run multiple parallel queues?

This question suggested that there are named queues, but do these all run off the one table and are thus sequential?

At the bottom @Jesse Wolgamott suggests that you can create a table for each queue that will then run in parrallel.

Has anyone done this and can they point me to how it is done?

Upvotes: 20

Views: 15039

Answers (3)

Ggmon
Ggmon

Reputation: 748

It is possible and I am doing it all the time. In our case we need multiple jobs for processing three different kinds of jobs say queue_a, queue_b, and queue_c. The system will make entries in the delayed_job table and the queue named appropriately.

Start multiple delayed jobs like

RAILS_ENV=production script/delayed_job -i first --queue=queue_a start
RAILS_ENV=production script/delayed_job -i second --queue=queue_a start
RAILS_ENV=production script/delayed_job -i third --queue=queue_b  start
RAILS_ENV=production script/delayed_job -i fourth --queue=queue_c start

Each command will create a delayed_job, so there will be now 4 parallel jobs, two of them serving queue_a and one each for queue_b and queue_c. The key thing here is the identifier that is passed through the -i option which specifies the instance name, and we can start and stop jobs as required.

Another option is to use worker pools.

RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start

That command will start 1 worker for the tracking queue, 2 workers for the mailers and tasks queues, and 2 workers for any jobs.

Upvotes: 33

changingrainbows
changingrainbows

Reputation: 2711

If you're using rake tasks as your job launch mechanism, the QUEUE environment variable can be used to launch different worker jobs separated by queue.

Example:

QUEUE=email rake jobs:work
QUEUE=build_data rake jobs:work

The QUEUES variable lets one dequeue from multiple DJ queues for a particular worker.

QUEUES=build_data,email rake jobs:work

Particularly useful if you're using a hosted/cloud environment (e.g. Heroku) that gives you limited access to launching scripts/jobs directly.

Upvotes: 4

slotishtype
slotishtype

Reputation: 2751

With bundler in production:

RAILS_ENV=production bundle exec script/delayed_job -n 4 start

or without bundler

ruby script/delayed_job -n 4 start

Upvotes: 16

Related Questions