CODEWITHSUNDEEP
seleniumrspeccapybaracapybara-webkitdatabase-cleaner
Greg Blass
Greg Blass

Reputation: 3660

RSpec/Capybara/JS tests pass in isolation, but fail when run together

My issue seems to be very similar to: rspec test passes in isolation, but fails when run with other tests - but the accepted answer is something I'm already doing: I'm using avdi's database cleaner setup (and I've tried deletion instead of truncation like thoughtbot uses in their suspenders example app): http://devblog.avdi.org/2012/08/31/configuring-database_cleaner-with-rails-rspec-capybara-and-selenium/

So the account record from this code:

let(:account) { create(:account) }

seems to be being deleted. I'm hitting a page not found because the account doesn't exist. So switching back and forth between the different database cleaner strategies seems to be causing the issue?

I can literally run rspec --only-failures and all my tests that just failed will pass. Its only happening with feature/capybara specs.

EDIT: Here's my config: https://gist.github.com/gregblass/b886f79b0d8e2e2015af

UPDATE: Here are some examples of failing tests: https://gist.github.com/gregblass/1b878d92a2b9dad355e0

UPDATE 2: It happening after I do a js: true test. Something with those are screwing up my other non-js capybara tests that follow it.

Upvotes: 2

Views: 2124

Answers (1)

Greg Blass
Greg Blass

Reputation: 3660

I was able to use rspec --bisect and pin down the culprits - they were JS capybara tests. After I ran JS tests, the database cleaner switching strategy that everyone seems to recommend is deleting my accounts that I'm declaring in my non-js capybara tests that come after.

This is one workaround:

http://stefan.magnuson.co/articles/rails/robust-integration-testing-in-rails-4-with-rspec-capybara-and-selenium/

Many tutorials and StackOverflow answers out there have JS and non-JS tests mixed together, with the strategy being switched between transaction and truncation on each test.

In my experience this has resulted in database access race conditions that have caused otherwise well-written, independent tests to fail intermittently.

The solution I have settled on is to run all non-JS tests first (shuffled), and then all JS tests (also shuffled). This allows for discovery of tests that incorrectly expect certain state, or fail to clean up after themselves (by virtue of the random execution order), while not attempting to freely mix JS and non-JS tests. As these different classes of test have different purposes, I see no disadvantage in this approach, with the exception of it being somewhat non-standard.

Another is to change your database cleaner configuration to the following (taken from @ryanbigg 's multitenancy with rails book):

config.before(:suite) do
  DatabaseCleaner.strategy = :truncation
  DatabaseCleaner.clean_with(:truncation)
end

config.around(:each) do |example|
  DatabaseCleaner.cleaning do
    example.run
  end
end

This is slower, but I can't even really tell much of a difference on my Macbook Pro 16GB/SSD.

Thank you @tomwalpole for all your help, @barelyknown for suggesting rspec --bisect, and @ryanbigg for your great book on testing multitenancy with capybara with good working code examples, and Stefan Magnuson for that article (who I can't find a SO handle for).

Upvotes: 5

Related Questions