CODEWITHSUNDEEP
ruby-on-railsherokuqueueresqueworker
Jérôme Boé
Jérôme Boé

Reputation: 2938

Autoscaling Resque workers on Heroku in real time

I would like to up/down-scale my dynos automatically dependings on the size of the pending list.

I heard about HireFire, but the scaling is only made every minutes, and I need it to be (almost) real time.

I would like to scale my dynos so that the pending list be ~always empty.

I was thinking about doing it by myself (with a scheduler (~15s delay) and using Heroku API), because I'm not sure there is anything out there; and if not, do you know any monitoring tools which could send an email alert if the queue lenght exceed a fixed size ? (similar to apdex on newrelic).

Upvotes: 1

Views: 498

Answers (2)

jalagrange
jalagrange

Reputation: 2391

Im having a similiar issue and have ran into "Hirefire"

https://www.hirefire.io/.

For ruby, use:

https://github.com/hirefire/hirefire-resource

It runs similar to theoretically works like AdepScale (https://www.adeptscale.com/). However Hirefire can also scale workers and does not limit itself to just dynos. Hope this helps!

Upvotes: 0

Lukas Eklund
Lukas Eklund

Reputation: 6138

A potential custom code solution is included below. There are also two New Relic plgins that do Resque monitoring. I'm not sure if either do email alerts based on exceeding a certain queue size. Using resque hooks you could output log messages that could trigger email alerts (or slack, hipchat, pagerduty, etc) via a service like Papertrail or Loggly. THis might look something like:

def after_enqueue_pending_check(*args)
  job_count = Resque.info[:pending].to_i
  if job_count > PENDING_THRESHOLD
    Rails.logger.warn('pending queue threshold exceeded')
  end
end

Instead of logging you could send an email but without some sort of rate limiting on the emails you could easily get flooded if the pending queue grows rapidly.

I don't think there is a Heroku add-on or other service that can do the scaling in realtime. There is a gem that will do this using the deprecated Heroku API. You can do this using resque hooks and the Heroku platform-api. This untested example uses the heroku platform-api to scale the 'worker' dynos up and down. Just as an example I included 1 worker for every three pending jobs. The downscale will only every reset the workers to 1 if there are no pending jobs and no working jobs. This is not ideal and should be updated to fit your needs. See here for information about ensuring that then scaling down the workers you don't lose jobs: http://quickleft.com/blog/heroku-s-cedar-stack-will-kill-your-resque-workers

require 'platform-api'

def after_enqueue_upscale(*args)
  heroku = PlatformAPI.connect_oauth('OAUTH_TOKEN')
  worker_count = heroku.formation.info('app-name','worker')["quantity"]
  job_count = Resque.info[:pending].to_i
  # one worker for every 3 jobs (minimum of 1)
  new_worker_count = ((job_count / 3) + 1).to_i
  return if new_worker_count <= worker_count
  heroku.formation.update('app-name', 'worker', {"quantity" => new_worker_count})
end

def after_perform_downscale
  heroku = PlatformAPI.connect_oauth('OAUTH_TOKEN')
  if Resque.info[:pending].to_i == 0 && Resque.info[:working].to_i == 0
      heroku.formation.update('app-name', 'worker', {"quantity" => 1})
  end
end

Upvotes: 1

Related Questions