CODEWITHSUNDEEP
ruby-on-railsherokuunicorn
imderek
imderek

Reputation: 1316

Error R12 (Exit timeout) using Heroku's recommended Unicorn config

My Unicorn config (copied from Heroku's docs):

# config/unicorn.rb
worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)
timeout 30
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end 

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

But every time a dyno is restarted, we get this:

heroku web.5 - - Error R12 (Exit timeout) -> At least one process failed to exit within 10 seconds of SIGTERM

Ruby 2.0, Rails 3.2, Unicorn 4.6.3

Upvotes: 23

Views: 3918

Answers (2)

bcb
bcb

Reputation: 939

I hate to add another answer, especially one this simple, but ultimately what fixed this problem for us was removing the 'rack-timeout' gem. I realize this is probably not best practice but I'm curious if there is some conflict between rack-timeout and Unicorn and/or Puma (which is odd because Heroku recommends rack-timeout for use with Unicorn).

Anyway Puma is working great for us but we did still see some random inexplicable timeouts even after the Puma upgrade . . . but removing rack-timeout got rid of the issue completely. Obviously we still get timeouts but only for code we haven't optimized or if we are getting heavy usage (basically when you would expect to see timeouts). Thus I would blame this issue on rack-timeout and not on Unicorn . . . thus contradicting my previous answer :)

Hope this helps. If anyone else wants to poke holes in my theory, feel free!

Upvotes: 6

bcb
bcb

Reputation: 939

We've had issues like this with Unicorn for some time . . . we also get seemingly random timeout errors, even though we never see much load and have 4 dynos with 4 workers each (we never have any request queuing). We have had 0 luck getting rid of these errors, even with help from Heroku. I get the feeling even they aren't 100% confident in the optimal settings for Unicorn on Heroku.

We just recently switched to Puma and so far so good, much better performance and no weird timeouts yet. One of the other reasons we switched to Puma is that I suspect some of our random timeouts come from "slow clients" . . . Unicorn isn't designed to handle slow clients.

I will let you know if we see continued success with Puma, but so far so good. The switch is pretty painless, assuming your app is thread-safe.

Here are the puma settings we are using. We are using "Clustered Mode".

procfile:

web: bundle exec puma -p $PORT -C ./config/puma.rb

puma.rb:

environment ENV['RACK_ENV']
threads Integer(ENV["PUMA_THREADS"] || 5),Integer(ENV["PUMA_THREADS"] || 5)

workers Integer(ENV["WEB_CONCURRENCY"] || 4)
preload_app!

on_worker_boot do
  ActiveSupport.on_load(:active_record) do
    ActiveRecord::Base.establish_connection
  end
end

We currently have WEB_CONCURRENCY set to 4 and PUMA_THREADS set to 5.

We aren't using an initializer for DB_POOL, just using the default DB_POOL setting of 5 (hence the 5 threads).

The only reason we are using WEB_CONCURRENCY as our environment variable name is so that log2viz reports the correct number of workers. Would rather call it PUMA_WORKERS but whatever, not a huge deal.

Hope this helps . . . again, will let you know if we see any issues with Puma.

Upvotes: 10

Related Questions