Sudden inexplicable active record connection timeouts while testing with RSPEC

Question

This is an area I know almost nothing about, so apologies in advance. I have a suite of over 800 rspec tests. Suddenly and inexplicably when running the whole set or just particular test files, after just a few of these, (say 20 or so, although it's never exactly the same number), every single test begins to fail with the same error:

 Failure/Error: Unable to find matching line from backtrace
 ActiveRecord::ConnectionTimeoutError:
   could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)

In a typical run, I'll start getting these errors after 20 or so request tests, and the remaining 780+ tests all fail with the exact same error above. I've tried going back to a previous git commit and branch that previously tested perfectly. No luck — still 780+ failures. Also completely dropped an recreated test DB. Also no luck.

I've read many threads about connection pools etc., but I'm afraid I have no idea how to diagnose even what's going on. Here are the facts as I know them right now:

• Using Postgresql
• Development environment works fine as far as I can tell
• In between the time everything was working fine and now, I've made no changes/upgrades to the environment that I'm aware of. Not even any database migrations. Merely changes to model, view and controller code. And as I mentioned, going back to previous commits doesn't fix anything.
• config.use_transactional_fixtures = false in spec_helper.rb because I'm testing ajax functionality via Selenium.
• Tests, however, fail regardless of whether Selenium is being used for the particular set of tests. Even if I run only tests that are not using Selenium, failures still start after 20 or so tests.

Instead of transactional fixtures, I am using Database Cleaner with the following configuration:

  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
  end

  config.before(:each, :js => true) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end

Any ideas what's going on here? And more specifically, any ideas where I should be looking to see what the problem is? I have almost no experience dealing with ActiveRecord issues of this type and I don't even know where to start.

Update While I still don't know exactly why this is happening, I do know specifically what code is causing it. In a recent commit I had added sending of a notification email in a new thread. Here's the code:

    def teacher_notification_email
      Thread.new do
        UserMailer.accepted_parent_invitation_email(@parent_profile).deliver
        ActiveRecord::Base.connection.close
      end
    end

I've used this exact pattern (with different emails) in many other places in the application, all of which get tested. For some reason, this particular one is causing database timeout errors. Any ideas on why this is happening are welcome.

Update Since I'm not at a stage where I understand how threading works in this instance, I don't know the exact source of the problem, other than this: from what I've read, it's very hard to programmatically control the execution of a thread created in this way. However, I've found a solution. Instead of the above block, I've changed the block to the following:

    def teacher_notification_email
      if Rails.env.test?
        UserMailer.accepted_parent_invitation_email(@parent_profile).deliver
      else
        Thread.new do
          UserMailer.accepted_parent_invitation_email(@parent_profile).deliver
          ActiveRecord::Base.connection.close
        end
      end
    end

So I'm basically running different code for test than development - no new thread for test. I'm assuming that's a bad idea, but until I can understand where the real problem is (a test that doesn't inherently fail for whatever reason, or code that still uses the thread that doesn't force a failed test), this is what I need to go with.

Final Update

I've dropped the Thread.new method of asynchronously sending emails and have, instead, implemented Sidekiq. It's a little more work, but it works well and tests fine...

Answer 1

It seems this is related to touching active record inside a spawned thread. It looks like the db connection doesn't get returned to the pool until it is reaped. I've been able to resolve this issue by explicitly asking for a connection ahead of time and closing it after I'm done. Try this:

Thread.new do
    ActiveRecord::Base.connection_pool.with_connection do |conn|
        UserMailer.accepted_parent_invitation_email(@parent_profile).deliver
    end
end