Rails Rspec Suite Has Failures But They Pass When Run Individually

Question

Facts:

1. Running my entire suite of specs will result in 21 consistent errors out of 610 specs.
2. If I run any individual spec file (for instance: messages_controller_spec.rb), they will all pass.
3. If I run any of the failed specs individually, they will each pass.

These errors are mostly ActionMailer failures, but some are otherwise. One confusing aspect is that some of the specs fail because there was an extra row in the database than expected, while others fail because there was one less row than expected. That is, if it were a cleaning or cache issue, it seems like it should be consistently one more or one less.

I’m currently running Rails 4.1.1, Ruby 2.0.0p451, Rspec 2.14.8, Sidekiq via inline!

Gemfile(for test)

group :development, :test do
  gem 'better_errors'
  gem 'binding_of_caller'
  gem 'faker'
  gem 'guard-rspec'
  gem 'pry'
  gem 'rspec-rails'
  gem 'spork-rails'
  gem 'sqlite3'
  gem 'thin'
end

group :test do
  gem 'capybara'
  gem 'capybara-email'
  gem 'capybara-webkit'
  gem 'database_cleaner'
  gem 'fabrication'
  gem 'launchy'
  gem 'selenium-webdriver'
  gem 'shoulda-matchers'
  gem 'webmock'
  gem 'vcr'
end

Please note, I’ve checked out about a dozen similar questions that did not work to fix this. So, to clarify:

1. I’m not using ARGV
2. I’m not using before(:all) - I use before(:each).
3. I have attempted Rails.cache.clear before both the :suite & the :each spec.
4. I have database cleaner set to :truncation for all cleaning (slower, but better results than :transaction - have had issues with reloading updated values using :transaction)

For convenience sake, maybe some examples will help:

scheduler_spec.rb (showing test that fails when whole suite is run)

require 'spec_helper'
require 'rake'
require 'sidekiq/testing'
Sidekiq::Testing.inline!

describe "scheduler", :vcr do
  describe ":wipe_abandoned_images" do
    let!(:abandoned_old_image)   { Fabricate(:image) }
    let!(:abandoned_young_image) { Fabricate(:image) }
    let!(:adopted_image)         { Fabricate(:image) }
    let(:run_cleaner) do
      Rake::Task[:wipe_abandoned_images].reenable
      Rake.application.invoke_task :wipe_abandoned_images
    end

  before do
      abandoned_young_image.update_columns(listing_id: nil, updated_at: 6.days.ago)
      abandoned_old_image.update_columns(  listing_id: nil, updated_at: 9.days.ago)
      Rake.application.rake_require 'tasks/scheduler'
      Rake::Task.define_task(:environment) #Stub env. Rspec runs the App, so dont want Rake to run it again.
  end

  context "for claimed images" do
      it "leaves the image" do
        adopted_image_id = adopted_image.id
        run_cleaner
          expect(Image.all.count                ).to eq(2)
          expect(Image.find(adopted_image_id) ).to be_present
      end
    end
  end
end

Note that I'm using Sidekiq's inline! testing configuration on prior apps with good success & without this issue.

forgot_passwords_controller_spec.rb (showing test that fails when whole suite is run)

require 'spec_helper'
require 'sidekiq/testing'
Sidekiq::Testing.inline!

describe ForgotPasswordsController do
  let!(:jen) { Fabricate(:user, email: 'jen@example.com') }

  describe "POST create" do
    context "with valid email provided" do
      before { post :create, email: 'jen@example.com' }
      after do
        ActionMailer::Base.deliveries.clear
        Sidekiq::Worker.clear_all
      end

      it 'sends the reset email to the users provided email' do
        expect(ActionMailer::Base.deliveries.count).to eq(1)
      end
    end
  end
end

Here's what happens when I run the specs in various ways:

ForgotPasswordsController Specs pass via RubyTest plugin in SublimeText2

2014-08-28T03:42:43Z 32968 TID-ov5g65p44 INFO: Sidekiq client with redis options {} ........... Finished in 0.95479 seconds 11 examples, 0 failures Randomized with seed 40226 [Finished in 5.8s]

Scheduler Tests pass via RubyTest in SublimeText2

.Rationing out invitations to users... done. .Rationing out invitations to users... done. ..Sweeping the server for abandoned images... 2014-08-28T01:49:02Z 32426 TID-owjt9ggh8 INFO: Sidekiq client with redis options {} done. .Sweeping the server for abandoned images... done. .Sweeping the server for abandoned images... done. .

Finished in 1.52 seconds 7 examples, 0 failures Randomized with seed 37996 [Finished in 8.6s]

Tests pass via rspec in Console $ rspec ./spec/lib/tasks/scheduler_spec.rb

.Rationing out invitations to users... done. .Rationing out invitations to users... done. ..Sweeping the server for abandoned images... 2014-08-28T02:14:43Z 32456 TID-ouiui9g8c INFO: Sidekiq client with redis options {} done. .Sweeping the server for abandoned images... done. .Sweeping the server for abandoned images... done. .

Finished in 1.32 seconds 7 examples, 0 failures Randomized with seed 19172

Tests fail when run entire suite of Rspec These failures will pass if run individually or as the spec file they are from. $ rspec

Finished in 49.71 seconds 610 examples, 21 failures, 10 pending

Failed examples:

Sometimes it has an extra value

13) scheduler :wipe_abandoned_images for abandoned images under 1 week old leaves the image Failure/Error: expect(Image.all.count ).to eq(2)

   expected: 2
        got: 3

   (compared using ==)
 # ./spec/lib/tasks/scheduler_spec.rb:78:in `block (4 levels) in <top (required)>'

Sometimes it loses or doesn’t load a value

18) ForgotPasswordsController POST create with valid email provided sends the reset email to the users provided email Failure/Error: expect(ActionMailer::Base.deliveries.count).to eq(1)

   expected: 1
        got: 0

   (compared using ==)
 # ./spec/controllers/forgot_passwords_controller_spec.rb:22:in `block (4 levels) in <top (required)>'

Here's a list of the failures

rspec ./spec/controllers/messages_controller_spec.rb:161 # MessagesController POST create message about listing to user from guest with valid information with EXISTING, UN-confirmed guest with EXPIRED token sends another confirmation email with link to the guest
rspec ./spec/controllers/messages_controller_spec.rb:114 # MessagesController POST create message about listing to user from guest with valid information with NEW, UN-confirmed, and valid guest email sends an invitation for the guest to be put on safe-email list
rspec ./spec/controllers/invitations_controller_spec.rb:30 # InvitationsController POST create with valid email & available invitations sends an email
rspec ./spec/controllers/invitations_controller_spec.rb:33 # InvitationsController POST create with valid email & available invitations sends an email to the recipient_email address
rspec ./spec/controllers/users_controller_spec.rb:161 # UsersController POST create with invitation token in params with valid token & input confirmation email sending sends the email to the registering user
rspec ./spec/controllers/users_controller_spec.rb:158 # UsersController POST create with invitation token in params with valid token & input confirmation email sending sends the email
rspec ./spec/controllers/users_controller_spec.rb:164 # UsersController POST create with invitation token in params with valid token & input confirmation email sending sends an email with a confirmation link in the body
rspec ./spec/controllers/users_controller_spec.rb:354 # UsersController GET confirm_with_token with valid token has a welcome message in the email
rspec ./spec/controllers/users_controller_spec.rb:348 # UsersController GET confirm_with_token with valid token sends a welcome email
rspec ./spec/controllers/users_controller_spec.rb:351 # UsersController GET confirm_with_token with valid token sends the welcome email to the user
rspec ./spec/controllers/searches_controller_spec.rb:19 # SearchesController GET search GET search with specific category selected returns the matching OR partial-matching table row objects
rspec ./spec/controllers/searches_controller_spec.rb:22 # SearchesController GET search GET search with specific category selected only returns values from the selected category
rspec ./spec/lib/tasks/scheduler_spec.rb:75 # scheduler :wipe_abandoned_images for abandoned images under 1 week old leaves the image
rspec ./spec/lib/tasks/scheduler_spec.rb:68 # scheduler :wipe_abandoned_images for abandoned images over 1 week old deletes the images
rspec ./spec/lib/tasks/scheduler_spec.rb:84 # scheduler :wipe_abandoned_images for claimed images leaves the image
rspec ./spec/controllers/forgot_passwords_controller_spec.rb:24 # ForgotPasswordsController POST create with valid email provided sets the email subject to notify the user of the reset link
rspec ./spec/controllers/forgot_passwords_controller_spec.rb:27 # ForgotPasswordsController POST create with valid email provided sends the link with token in the body of the email
rspec ./spec/controllers/forgot_passwords_controller_spec.rb:21 # ForgotPasswordsController POST create with valid email provided sends the reset email to the users provided email
rspec ./spec/controllers/reset_passwords_controller_spec.rb:70 # ResetPasswordsController POST create with a valid token sets the email subject to notify the user of the reset password
rspec ./spec/controllers/reset_passwords_controller_spec.rb:67 # ResetPasswordsController POST create with a valid token sends a confirmation email to the user that their password has been changed
rspec ./spec/controllers/reset_passwords_controller_spec.rb:73 # ResetPasswordsController POST create with a valid token sends the link with token in the body of the email

Answer 1

I can't explain what is really causing the error. But it must due to setting the Sidekiq test mode globally. Remove the Sidekiq setting from the head section of the specs and try the following:

    before do 
    Sidekiq::Testing.inline! do
      post :create, email: 'jen@example.com' 
     end  
    end
    after do
      ActionMailer::Base.deliveries.clear
      Sidekiq::Worker.clear_all
    end

    it 'sends the reset email to the users provided email' do
      expect(ActionMailer::Base.deliveries.count).to eq(1)
    end