CODEWITHSUNDEEP
herokuresqueunicorn
Viet
Viet

Reputation: 3387

Concurrency with Resque on Heroku (running multiple workers per node)

Pardon my ignorance, but is there to increase the number of processes per dyno for Resque workers? And if so, how?

I'm currently using Unicorn to add concurrency to the web dynos, which has been working great so far. I would like to extend this to the Resque workers. I followed Heroku's guide to set up the concurrency.

Upvotes: 3

Views: 964

Answers (1)

Lukas Eklund
Lukas Eklund

Reputation: 6138

Update: The solution below works, but is not recommended. For resque concurrency on heroku use the resque-pool gem.

It is possible if you use the COUNT=* option. Your procfile will look something like:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
resque: env TERM_CHILD=1 COUNT=2 RESQUE_TERM_TIMEOUT=6 QUEUE=* bundle exec rake resque:workers

It is impotant to note that the rake task in the Procfile is resque:workers and not resque:work.

Update Explanation

There are major problems with the COUNT=* option and the rake resque:workers invocation in production on heroku. Because of the way resque starts up the multiple workers using threads, all of the SIGTERM, SIGKILL, etc. handling that allows workers to stop the current job, re-enqueue job, and shut down properly (including de-registering) will ever happen. This is because the the signals are handled by the main process not trapped by the threads. This can cause phantom workers to remain in the worker list long after they've been killed. This is probably why there is a comment in the resque code that warns that the resque:workers should only be used in development mode.

Upvotes: 5

Related Questions