How to properly test ActiveJob's retry_on method with rspec?

Question

I have been attempting to test this method for the past few days with no luck.

Another thing I'd like to be able to do is rescue the error that bubbles up after the final retry attempt is made.

Please see my comments and code snippets below.

Source code for retry_on is here as well for context.

Here's the sample code and tests:

   my_job.rb

   retry_on Exception, wait: 2.hours, attempts: 3 do |job, exception|
   # some kind of rescue here after job.exceptions == 3  
   # then notify Bugsnag of failed final attempt.
   end

   def perform(an_object)
     an_object.does_something
   end

   my_spec.rb
   it 'receives retry_on 3 times' do
     perform_enqueued_jobs do
       expect(AnObject).to receive(:does_something).and_raise { Exception }.exactly(3).times
       expect(MyJob).to receive(:retry_on).with(wait: 2.hours, attempts: 3).exactly(3).times
       MyJob.perform_later(an_object)
     end
     assert_performed_jobs 3
   end

The test failure response:

      1) MyJob.perform receives retry_on 3 times
         Failure/Error: expect(job).to receive(:retry_on).with(wait: 4.hours, attempts: 3).exactly(3).times

   (MyJob (class)).retry_on({:wait=>2 hours, :attempts=>3})
       expected: 3 times with arguments: ({:wait=>2 hours, :attempts=>3})
       received: 0 times
 # ./spec/jobs/my_job_spec.rb:38:in `block (4 levels) in <top (required)>'
 # ./spec/rails_helper.rb:48:in `block (3 levels) in <top (required)>'
 # ./spec/rails_helper.rb:47:in `block (2 levels) in <top (required)>'

I've also tried making the job a double and stubbing the retry_on method and that doesn't work either.

I've also tried using Timecop to fast forward the wait time and tests are still failing:

           my_spec.rb
   it 'receives retry_on 3 times' do
     perform_enqueued_jobs do
       expect(AnObject).to receive(:does_something).and_raise { Exception }.exactly(3).times
       Timecop.freeze(Time.now + 8.hours) do
         expect(MyJob).to receive(:retry_on).with(wait: 2.hours, attempts: 3).exactly(3).times
       end
       MyJob.perform_later(an_object)
     end
     assert_performed_jobs 3
   end

It IS a class method of ActiveJob and I've confirmed this in a byebug terminal that this is the case with my job class.

Shouldn't this test work? It's expecting the class to receive the class method with certain arguments. My byebug gets hit when I put it in the retry_on block as well so I know that the method is getting called multiple times.

It's almost as if it's being called on a different class which is very confusing and I don't think is the case but I'm at the end of my rope with this one.

I almost resolved the issue by decoupling my tests from testing the retry_on rails logic itself to testing my business logic around it. This way is better as well in the case that rails ever changes the retry_on logic.

HOWEVER, this does NOT work for more than one test case. If you use this with more than one case, the last test will break and say it has performed more jobs than expected.

 my_spec.rb
 it 'receives retry_on 3 times' do
   perform_enqueued_jobs do
     allow(AnObject).to receive(:does_something).and_raise { Exception }
     expect(AnObject).to receive(:does_something).exactly(3).times
     expect(Bugsnag).to receive(:notify).with(Exception).once
     MyJob.perform_later(an_object)
   end
   assert_performed_jobs 3
 end

my_job.rb

retry_on Exception, wait: , attempts: 3 do |job, exception|
  Bugsnag.notify(exception)
end

def perform(an_object)
  an_object.does_something
end

Any help/insight on this would be greatly appreciated.

Would also love a recommendation on how to handle the bubbled up exception after max attempts too. I'm thinking of raising an error within the retry_on block and then have discard_on trigger for the error that's raised.

Thank you wonderful Stack Overflow community!

Answer 1

My solution for minitests:

class ExecuteJob < ApplicationJob
  RETRY_LIMIT = 30
  RETRY_DELAY = 1.second

  retry_on ExpectedErrorClass, wait: RETRY_DELAY, attempts: RETRY_LIMIT

  def perform
   # your code
  end
end

test 'retries run job' do
  assert_performed_jobs(described_class::RETRY_LIMIT) do
    perform_enqueued_jobs do
      assert_raises ExpectedErrorClass do
        described_class.perform_later
      end
    end
  end
end

Answer 2

While I agree with @fabriciofreitag that one shouldn't need to test the internals of an external library, I think there's definitely value in confirming that your retry_on blocks are configured properly. This setup worked for me without having to worry about how ActiveJob manages the retries.

# app/jobs/my_job.rb
retry_on TimeoutError,
         wait: :exponentially_longer,
         attempts: 3

# spec/jobs/my_job_spec.rb
describe "error handling" do
  before do
    ActiveJob::Base.queue_adapter.perform_enqueued_at_jobs = true
  end

  context "when TimeoutError is raised" do
    it "retries timed-out requests" do
      expect(client).to receive(:connect).ordered.and_raise(TimeoutError)
      expect(client).to receive(:connect).ordered.and_call_original
      described_class.perform_now
    end

    it "retries three times before re-raising" do
      expect(client).to receive(:connect)
        .exactly(3).times.and_raise(TimeoutError)
      expect { described_class.perform_now }.to raise_error(TimeoutError)
    end
  end
end

Answer 3

IMHO you should leave the testing of ActiveJob with the rails team.

You only need to make sure you're configuring the job properly:

it 'retries the job 10 times with 2 minutes intervals' do
  allow(MyJob).to receive(:retry_on)
  load 'app/path/to/job/my_job.rb'
  expect(MyJob).to have_received(:retry_on)
    .with(
      Exception,
      wait: 2.minutes,
      attempts: 10
    )
end

Answer 4

The following works fine for me, also for multiple testcases and for testing side effects of the retry_on block.

RSpec.describe MyJob, type: :job do
  include ActiveJob::TestHelper

  context 'when `MyError` is raised' do
    before do
      allow_any_instance_of(described_class).to receive(:perform).and_raise(MyError.new)
    end

    it 'makes 4 attempts' do
      assert_performed_jobs 4 do
        described_class.perform_later rescue nil
      end
    end

    it 'does something in the `retry_on` block' do
      expect(Something).to receive(:something)

      perform_enqueued_jobs do
        described_class.perform_later rescue nil
      end
    end
  end
end

Note that rescue nil (or some form of rescue) is required if you let exceptions bubble up at the end.

Note that perform_now doesn't count as "enqueued job". So doing described_class.perform_now results in one less attempts counted by assert_performed_jobs.

Answer 5

This is the format of specs needed for retry_on that finally worked for me:

it 'receives retry_on 10 times' do
  allow_any_instance_of(MyJob).to receive(:perform).and_raise(MyError.new(nil))
  allow_any_instance_of(MyJob).to receive(:executions).and_return(10)
  expect(Bugsnag).to receive(:notify)
  MyJob.perform_now(an_object)
end

it 'handles error' do
  allow_any_instance_of(MyJob).to receive(:perform).and_raise(MyError.new(nil))
  expect_any_instance_of(MyJob).to receive(:retry_job)
  perform_enqueued_jobs do
    MyJob.perform_later(an_object)
  end
end

For the first case, executions is an ActiveJob method that gets run, set and checked every time retry_on is executed. We mock it to return 10 and then expect it to call Bugsnag. retry_on only calls what you gave it in the block once all the attempts have been met. So this works.

For the second case, Then mock the error to raise for the job instance. Next we check that it's correctly receiving retry_job (which retry_on calls under the hood) to confirm it's doing the right thing. Then we wrap the perform_later call in the minitest perform_enqueued_jobs block and call it a day.

Answer 6

In the first spec

expect(MyJob).to receive(:retry_on).with(wait: 2.hours, attempts:3).exactly(3).times

This will never gonna work since class method retry_on will be called on class initialization phase, ie when loading that class into memory, not while executing a spec

In the second spec, you tried to make it work using timecop but still failed for the same reason

Third spec is relatively more realistic, but

assert_performed_jobs 3

won't work without passing the block

Something like

assert_performed_jobs 2 do
  //call jobs from here
end