Rails 3.0.7 -> How do you get your tests to run faster?

Question

I am running mysql, database_cleaner, Rspec, etc. I have about 518 tests so far and they take 88 seconds to run. This is unacceptable to me as my app development is just beginning.

So before going further, I'd like to try and find ways to reduce the time it takes to run these tests - hopefully without having to actually change the tests.

In most cases, I am trying to use stubs. However, when I am testing models and queries, I do use the database.

I think database_cleaner is slowing them down, but I don't know how to test queries and stuff without it.

Using sqlite3 with the ":memory:" option only seems to shave off about 10 seconds (kind of disappointing result...)

What can I do to really speed up my tests?

Answer 1

Ryan Brunner offered a lot of great advice. Everything he said is true in general, yet did not apply to me.

I didn't mention Factory Girl because I didn't think to mention it (don't ask). It turned out to be very relevant detail because it was responsible for the tests running so slow.

By simply removing Factory girl completely from my controller tests (I was using Factory.build), I have managed to get them down from 50 seconds to something like 5.

The reason is that Factory.build calls Factory.create for associations, which causes a database hit... so if you have a lot of associations, it will take awhile to create a new model object. But even more, that only accounted for 30-35% of the overhead in my case. Factory_girl was actually spending 65-70% of its time doing non-database stuff. I have no idea why, but after forcing every call to be Factory.build, it will still taking quite awhile to build my objects. Going with basic MyClass.new ended up being MUCH faster.

My entire test suite now takes a little under 30 seconds instead of up to 90 seconds. That is a 300% speed increase in general by making these changes... but when it came to the controller tests, I got a 2000% speed increase - and I was already stubbing! All of that performance overhead was due to Factory.build! That is where most of the gains came from.

Of course, I went back into my models and used Factory.build or simply MyClass.new wherever I could.

I also added :default_strategy => :build in factories.rb too whenever I could, to prevent Factory Girl from hitting the database. If you ask me, this should be the default as only 1 test failed as a result of this change, but I managed to get 10 entire seconds out of my model tests by this change alone.

If you're having problems like I am, follow these steps and you should notice a 2-3x speed improvement with not much drawback.

Answer 2

I think that if you enable use_transactional_fixtures in RSpec then you shouldn't need to use database_cleaner at all.

Also, consider using NullDB to avoid hitting the real database, except where you have to (my approach is to never hit the database in unit tests, only in integration tests).

Answer 3

Look at parallel testing with Hydra:

http://logicalfriday.com/2011/05/18/faster-rails-tests-with-hydra/

Look here also:

http://www.adomokos.com/2011/04/running-rails-rspec-tests-without-rails.html

Answer 4

There's a variety of strategies you can use to speed up your test times. If you're just starting, and you're seeing an 88 second run time, I would imagine a good number of these apply to you:

• Use spork - Spork will do all the bootstrapping and environment requires once, keep that in memory, and only reload your tests. It can be a huge help with running tests quickly.
• Be smart about how you test - Personally, my workflow is to develop tests, run the full test suite to see what fails, and then only run the new / failed tests until I can get those green. Finally, when I'm all done I run the suite one last time to see if I regressed something else.
• Clean up your Gemfile - The majority of Rails boot time is spent in requires, and if you have gems you're not using anymore, you're adding load time for no benefit. Take out anything you're not using, and consider placing things only used in one or two spots in a named group so you can require them manually during execution (be careful with this - you're trading initial load speed for request performance, which is great for dev, but crappy for production)
• Be smart about what to stub and mock out - If you're truly doing unit tests, for instance, you should avoid touching the database althogether, or at least for your controller tests. Think about what the responsibility of the class really is. A controller isn't responsible for saving records, it's responsible for telling the models to save records. Even models aren't responsible for saving things in databases, they're responsible for telling ActiveRecord to.
• If you're stubbing things out well, consider not including Rails. You'll need to have nearly all ActiveRecord functionality stubbed out in your tests, but if you can do this, you'll see a massive decrease in your test time (probably more than an order of magnitude).

Answer 5

I've been using the following hack to reduce time spent in the Garbage Collector:

http://makandra.com/notes/950-speed-up-rspec-by-deferring-garbage-collection

The article mentions a 15% improvement, but in my tests, I'm seeing around 25% with Ruby 1.9.2, Rails 3.0.x and RSpec 2.0.

Also, if you aren't using autotest, that may help, so you are only running tests for code that has changed.

Finally, try using the RSpec "--profile" option to identify the 10 slowest examples, and see if you can optimize the performance of the worst offenders; In one of my projects it turned out that just 3 of my tests doubled my test execution time for 150 tests, so I "fixed" them and it brought the entire test suite back to an acceptable time scale.

Answer 6

Do you need to run all of your tests all of the time? You could set up rake tasks for different sets of tests and just run the ones that are relevant to parts of the application you've changed recently. Surely most of the tests would be running on code that hasn't changed. Then every so often you can run the full set of specs just to make sure everything's compatible. That seems like the easiest solution anyway.