CODEWITHSUNDEEP
ruby-on-railsperformanceruby-on-rails-5
DrewB
DrewB

Reputation: 3341

Big performance reduction after Rails 5 upgrade

We have completed upgrading our app from Rails 4.2 to 5.2. When we run load testing on the 5.2 version it can only handle half the load of the 4.2 version. In looking at NewRelic stats during the load tests it seems to be slower everywhere - pretty much every request, ActiveRecord calls, redis calls, ruby, etc. We have confirmed it is not related to other upgrades that happened in addition - ruby upgrade, upgrading pg gem, or upgrading puma. While researching, the only performance issues I have found related to the upgrade have been fixed.

Has anyone run into something similar or have pointers on where to look?

What we have tried so far:

1. Check non-rails related upgrades that happened at the same time:
   - Upgrade 4.2 branch to same version of ruby to see if that has any impact (no impact)
   - Downgrade puma and pg gems in Rails 5 branch (no impact)
2. Examine performance traces for slower transactions and DB queries.  Remove the slowest interactions from the load test to see if the overall slowest continues (it does).
3. Test if slowness appears in Rails 5.1 (it does).

What we are planning to try:

1. Test if slowness appears in Rails 5.0.
2. See if slowest can be detected in single user use rather than load test.
3. Use https://github.com/tleish/ruby-prof-rails to see if we can get more statistics to examine.
4. Downgrade all gems except the ones we absolutely need for the Rails 5 upgrade and see if problem still exists.

Upvotes: 4

Views: 716

Answers (1)

DrewB
DrewB

Reputation: 3341

This ended up being a combination of Rack::Timeout, Heroku and Puma. Under heavy load, we would sometimes hit our Rake::Timeout value of 28 seconds. For some reason after the upgrade, the 2 seconds between the Rake::Timeout value and Heroku's 30 second router timeout (H12) was not enough. As a result, processes were getting killed by Heroku before Rake::Timeou which was causing a cascade effect in Puma that made a ton of other requests on the same server also timeout. We fixed it by adjusting our Rake::Timeout value to 25 seconds and everything worked.

Upvotes: 1

Related Questions