CODEWITHSUNDEEP
rubymonitoringgod
bensie
bensie

Reputation: 5403

God - starting new process while existing process is still stopping

Using God (godrb.com) I'm trying to write a recipe that starts up a new process regardless of the status of an existing process when deploying an application. The existing process needs to have a long running timeout for it to finish current tasks, but the new process should start immediately using the newly deployed code.

What I currently have now sets a timeout of 300 seconds on the stop but waits the whole 300 seconds before starting up the new process.

God.watch do |w|
  w.name = "sidekiq"
  w.interval = 30.seconds
  w.start = "bash -lc 'cd /path/to/current && ./bin/sidekiq -P /path/to/shared/pids/sidekiq.pid'"
  w.stop  = "bash -lc 'kill -USR1 `cat /path/to/shared/pids/sidekiq.pid`'"
  w.stop_timeout = 300.seconds
  w.pid_file = "/path/to/shared/pids/sidekiq.pid"
  w.behavior(:clean_pid_file)
end

In this case, the kill -USR1 tells sidekiq to finish processing any current jobs, but to not take anymore work.

I'd like to keep the 300 second timeout on the existing worker but start up the new process as soon as the kill command is run.

Upvotes: 3

Views: 870

Answers (1)

scorix
scorix

Reputation: 2516

I think you have to define some transitions.

This is my god.rb:

# I'm using Rails
rails_env = ENV['RAILS_ENV'] || 'development'
rails_root = '/path/to/current'
pid_file = "#{rails_root}/tmp/pids/sidekiq.pid"

God.watch do |w|
  w.dir = rails_root
  w.name = "sidekiq"
  w.interval = 30.seconds
  w.env = {'RAILS_ENV' => rails_env, 'BUNDLE_GEMFILE' => "#{rails_root}/Gemfile"}
  w.uid = 'deployer'
  w.gid = 'staff'

  w.start = "cd #{rails_root}; bundle exec sidekiq -e #{rails_env} -C #{rails_root}/config/sidekiq.yml -i #{i} -P #{pid_file}&"
  w.stop = "cd #{rails_root}; bundle exec sidekiqctl stop #{pid_file} 10"
  w.restart = "#{w.stop} && #{w.start}"
  w.start_grace = 15.seconds
  w.restart_grace = 15.seconds
  w.pid_file = pid_file
  w.log = "#{rails_root}/log/sidekiq.log"

  # clean pid files before start if necessary
  w.behavior(:clean_pid_file)

  # determine the state on startup
  w.transition(:init, {true => :up, false => :start}) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end

    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_exits)
  end

  # lifecycle
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

Upvotes: 0

Related Questions