Rspec suite fails running on AWS Ubuntu with ECONNREFUSED - but each test passes individually

Question

We are trying to migrate our rails build from a local network server to an AWS EC2 instance running Ubuntu. The test suite works fine on all the developers machines (combination of mac and ubuntu) and the old build server (linux) but refuses to pass on the amazon instance.

The symptom is:

An error occurred in an after hook
   Errno::ECONNREFUSED: Connection refused - connect(2)
   occurred at /home/ubuntu/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http.rb:878:in 'initialize'

which is repeated a bunch of times. There is no error logged before this one.

However this issue only occurs when running multiple tests together (eg, bundle exec rspec) - when I run each test individually they all pass. Locally none of the dev team have seen this issue before. Our tests use Selenium and Firefox which should run headless on ubuntu using Xvfb. We also use WebMock for some tests.

I have researched this issue for a day or two now and tried everything I can think of, to no avail. I have checked the versions of firefox and all the gems installed are the same on aws as locally.

My gemfile and spec_helper are below. I have investigated the firefox and xvfb installation, but because each test is able to pass individually I feel the system is generally ok ... but clearly something is not happy.

Version details

ruby 2.0.0p481
rails 3.2.16
rspec 2.14.8
firefox 29
ubuntu 14.04

Gemfile

source 'https://rubygems.org'
#Base gems
gem 'rails', '~> 3.2.16'
gem 'sass-rails'
gem 'bcrypt-ruby', '~> 3.0.0'
gem 'faker', '~> 1.2.0'
gem 'jquery-rails', '~> 3.0.0'
#gem 'jquery-rails', '~> 2.1.4'
gem 'jquery-migrate-rails'

# threading gem
gem 'sucker_punch', '~> 1.0'
gem 'mixpanel-ruby'

# Mongo gems
gem 'mongoid', '~> 3.1.6'
gem 'bson_ext', '~> 1.9.2'
gem 'origin', '~> 1.1.0'
gem 'mongoid-enum' , '~> 0.1.1'

#filemaker gem
#gem 'ginjo-rfm', '~> 2.1.7'

# data gird gem
gem 'datagrid', '1.0.5'


# New Relic and Engine Yard Gems
gem 'ey_config', '~> 0.0.6'
gem 'newrelic_rpm'


#Authorsiation
gem 'cancan', '~> 1.6.10'

#paging
gem 'kaminari', '~> 0.15'

#client side validation
gem 'jquery-validation-rails'

#file upload
gem 'carrierwave-mongoid', :require => 'carrierwave/mongoid'
gem 'rmagick', '~> 2.13.2'
gem 'fog' , '~> 1.19.0'
gem 'unf', '~> 0.1.3'
gem 'remotipart', '~> 1.2'

gem 'listen', '2.7.6'

# reports for jenkins - see if having it here allows engineyard to deploy
gem 'ci_reporter', '1.9.1'

#NOTE YOU MUST INSTALL IMG MAGIC IF bundle install fails
#http://www.imagemagick.org/script/binary-releases.php#unix
#sudo apt-get install libmagickwand-dev imagemagick

gem 'timecop', '~> 0.7.1'

#file upload
gem 'jquery-fileupload-rails', '~> 0.4.1'

gem 'jquery-cookie-rails', '~> 1.3.1'

#configuring Rails
gem 'figaro'

#History tracking
gem 'mongoid-history'

gem 'nav_lynx'

#modal window requirements
gem 'jquery-ui-rails'
gem 'jquery-modal-rails'

gem 'itree'
gem 'lumberjack'

gem 'newrelic_moped'

group :development, :test do
  gem 'rspec-rails', '~> 2.14.0'
  gem 'guard-rspec', '~> 4.2.0'
  gem 'guard-spork', '~> 1.5.1'
  gem 'childprocess', '0.3.9'
  gem 'spork', '~> 1.0.0rc4'
  gem 'spork-rails', '~> 4.0.0'
  gem 'bullet'
end

group :development, :staging do
  gem 'rack-mini-profiler'
end

#Used to annotate model files generated
group :development do
  gem 'annotate', '~> 2.6.0'
  gem 'letter_opener', '~> 1.1.2'

end

# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'coffee-rails', '~> 3.2.2'
  gem 'uglifier', '~> 2.3.2'
end

group :test do
  gem 'capybara', '~> 2.2.1'
  gem 'selenium-webdriver', '2.39.0'
  gem 'database_cleaner', '~> 1.2.0'

  #needs sudo apt-get install xvfb
  gem 'headless'

  #static security analysis tool
  gem 'brakeman', :require => false

  #generating mock objects in tests
  gem 'factory_girl_rails', '~> 4.3.0'
  gem 'webmock', '~> 1.16.0'

  #for windows
  gem 'rb-fchange' if /win32/ =~ RUBY_PLATFORM
  gem 'rb-notifu' if /win32/ =~ RUBY_PLATFORM
  gem 'win32console' if /win32/ =~ RUBY_PLATFORM

  #for linux
  gem 'rb-inotify', '~> 0.9.2' if /linux/ =~ RUBY_PLATFORM
  gem 'libnotify', '~> 0.8.2' if /linux/ =~ RUBY_PLATFORM

  #for macosx
  gem 'growl', '~> 1.0.3' if /darwin/ =~ RUBY_PLATFORM
  gem 'rb-fsevent', '~> 0.9.3' if /darwin/ =~ RUBY_PLATFORM

  # for rcov in Jenkins
  gem 'simplecov'
  gem 'simplecov-rcov'

  #additional matches for rspect
  gem 'shoulda-matchers', '~> 2.5.0'

  #open browser on demand to debug tests
  gem 'launchy'

  #http  rest client
  gem 'rest-client', '~> 1.6.7'

  gem 'show_me_the_cookies'


end

spec_helper.rb

require 'rubygems'
require 'spork'
require 'webmock/rspec'
require 'sucker_punch'
require 'sucker_punch/testing/inline'

Spork.prefork do

  if ENV["RUBYMINE_HOME"]
    $:.unshift(File.expand_path("rb/testing/patch/common", ENV["RUBYMINE_HOME"]))
    $:.unshift(File.expand_path("rb/testing/patch/bdd", ENV["RUBYMINE_HOME"]))
  end

  # setup simple cov for Jenkins
  require 'simplecov'
  require 'simplecov-rcov'
  SimpleCov.formatter = SimpleCov::Formatter::RcovFormatter
  SimpleCov.start 'rails'

  ENV["RAILS_ENV"] ||= 'test'
  require File.expand_path("../../config/environment", __FILE__)
  require 'rspec/rails'
  require 'rspec/autorun'

  # Requires supporting ruby files with custom matchers and macros, etc,
  # in spec/support/ and its subdirectories.
  Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

  RSpec.configure do |config|

    config.include ShowMeTheCookies

    # Include Factory Girl syntax to simplify calls to factories
    config.include FactoryGirl::Syntax::Methods

    #Don't run any tests that contain :skip_test => true
    config.filter_run_excluding :skip_test => true

    # If true, the base class of anonymous controllers will be inferred
    # automatically. This will be the default behavior in future versions of
    # rspec-rails.
    config.infer_base_class_for_anonymous_controllers = false

    # Run specs in random order to surface order dependencies. If you find an
    # order dependency and want to debug it, you can fix the order by providing
    # the seed, which is printed after each run.
    #     --seed 1234
    config.order = "random"
    config.include Capybara::DSL

    config.before(:suite) do
      Headless.new.start
    end

    config.before(:all) do

      #manually clear database
      YearDates.destroy_all
      CoursePlan.destroy_all
      Topic.destroy_all
      FactoryGirl.create(:stub_year_dates)
    end

    config.after(:all) do
      Timecop.return
    end

    #Mongoid database cleaner
    config.before(:suite) do
      DatabaseCleaner[:mongoid, {:connection => :unit_test}].clean_with(:truncation)
      DatabaseCleaner[:mongoid, {:connection => :unit_test}].strategy = :truncation
    end

    config.before(:each) do
      new_time = Time.local(2014, 1, 20) #week two of the year
      Timecop.travel(new_time)
      Timecop.baseline = new_time
      DatabaseCleaner[:mongoid, {:connection => :unit_test}].start
      FactoryGirl.create(:stub_year_dates) #recreate this on each test as it is needed all the time, but edited sometimes
      #need to ensure these exist before the other factories are called
      FactoryGirl.create(:english_subject)
      FactoryGirl.create(:chemistry_subject)
      FactoryGirl.create(:maths_ext_2_subject)
      FactoryGirl.create(:maths_ext_1_tutorials_only_subject)
    end

    config.after(:each) do
      Timecop.return_to_baseline
      DatabaseCleaner[:mongoid, {:connection => :unit_test}].clean
    end

    Capybara.default_wait_time = 5 #wait longer for ajax requests
  end
end

Spork.each_run do
  # This code will be run each time you run your specs.
  FactoryGirl.reload

end

WebMock.allow_net_connect!

I would greatly appreciate any assistance!

update

I have determined that the results are inconsistent between test runs. At first I thought it was the same each time, but the number of tests failing varies each run. I thought I had fixed the problem when all the tests passed except one, but the next run it was back to over a hundred failures.

update 2

Upon closer inspection (oops) I have discovered there were some tests failing with a different error the the majority. Instead of ECONNREFUSED, they are reporting 'End Of File Reached'. When I disabled these tests (5 in total) the remaining tests pass reliably. I suspect that xvfb is encountering some javascript or ajax related error, which is then preventing any of the following tests from passing. We are now working on diagnosing what is happening in these tests that is causing the failure.