CODEWITHSUNDEEP
herokucronschedulerclockclockwork
oregontrail256
oregontrail256

Reputation: 657

Heroku clock process: how to ensure jobs weren't skipped?

I'm building a Heroku app that relies on scheduled jobs. We were previously using Heroku Scheduler but clock processes seem more flexible and robust. So now we're using a clock process to enqueue background jobs at specific times/intervals.

Heroku's docs mention that clock dynos, as with all dynos, are restarted at least once per day--and this incurs the risk of a clock process skipping a scheduled job: "Since dynos are restarted at least once a day some logic will need to exist on startup of the clock process to ensure that a job interval wasn’t skipped during the dyno restart." (See https://devcenter.heroku.com/articles/scheduled-jobs-custom-clock-processes)

What are some recommended ways to ensure that scheduled jobs aren't skipped, and to re-enqueue any jobs that were missed?

One possible way is to create a database record whenever a job is run/enqueued, and to check for the presence of expected records at regular intervals within the clock job. The biggest downside to this is that if there's a systemic problem with the clock dyno that causes it to be down for a significant period of time, then I can't do the polling every X hours to ensure that scheduled jobs were successfully run, since that polling happens within the clock dyno.

How have you dealt with the issue of clock dyno resiliency?

Thanks!

Upvotes: 13

Views: 1419

Answers (2)

Dan
Dan

Reputation: 108

Though it requires human involvement, we have our scheduled jobs check-in with Honeybadger via an after_perform hook in rails

# frozen_string_literal: true

class ScheduledJob < ApplicationJob
  after_perform do |job|
    check_in(job)
  end

  private

  def check_in(job)
    token = Rails.application.config_for(:check_ins)[job.class.name.underscore]
    Honeybadger.check_in(token) if token.present?
  end
end

This way when we happen to have poorly timed restarts from deploys we at least know should-be scheduled work didn't actually happen

Would be interested to know if someone has a more fully-baked, simple solution!

Upvotes: 0

Luc Boissaye
Luc Boissaye

Reputation: 945

You will need to store data about jobs somewhere. On Heroku, you don't have any informations or warranty about your code being running only once and all the time (because of cycling)

You may use a project like this on (but not very used) : https://github.com/amitree/delayed_job_recurring

Or depending on your need you could create a scheduler or process which schedule jobs for the next 24 hours and is run every 4 hours in order to be sure your jobs will be scheduled. And hope that the heroku scheduler will work at least once every 24 hours. And have at least 2 worker processing the jobs.

Upvotes: 1

Related Questions