How can I configure Delayed jobs to not wait for a task before starting the others?

Question

I am using Delayed jobs for my Ruby app hosted in Heroku to perform a very long task that can take up to 5 minutes. I've noticed that, in development mode at least, when this task is running the ones that come afterwards are not started until that one finishes. I would like other tasks to be able to start running without having to wait for the other to finish (to have at least 3 concurrent tasks, for example). I don't wish to increase the number of workers in Heroku ($$$). I noticed the 'pool' param in delayed jobs but I don't fully understand if this is what I need or how to use it.

https://github.com/collectiveidea/delayed_job/blob/master/README.md

I achieved it using threads in the task code, but maybe this is not the best way to do it.

If you could tell me exactly how I could achieve concurrency in delayed jobs I would really appreciate it.

Answer 1

A DJ worker only runs a single job at a time. If you want concurrent processing of your background jobs, you'll need multiple background workers.

You are way better off implementing sidekiq.