Memory exceeds in rails app in production (Deployed on Heroku)

Question

We've a rails app deployed on Heroku(RAM - 512mb). We've a API which returns 13k+ objects( Growing). Heroku app's memory crosses 200% of RAM. If i restart my app and hit this particular api 3-4 times, memory crosses RAM size immediately. Is it memory leak? Is it too much response time taking toll on memory?

 Controller code:
 def respond
  @us_anss= UsAns.respond_to_query(params)
 end

Model code

 def respond_to_query
 anss = self.all.includes(:a, :b, :c)
 anss.each do |ans|
    ans.class.class_eval {attr_accessor :coun, :date, :rec_id}
    ans.coun = ans.user.coun
    ans.date = ans.created_at.to_date
    if ans.rel_id.nil?
      ans.rec_id = nil
    else
      ans.rec_id = ans.user.opponent(ans.rel).id
    end
  end
  return anss
end

rabl code

Rail code
attributes :ans => :message_content, :user_id => :sen_id
attributes :rec_id, :coun, :date, :latitude, :longitude
collection @us_anss, :object_root => false
child :que do |us_wer|
   attributes :que => :text
   attributes :id, :tri_date
end

Gemfile

source 'https://rubygems.org'
# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '~> 5.0.0', '>= 5.0.0.1'
gem 'puma', '~> 3.0'
gem 'rabl'
# Also add either `oj` or `yajl-ruby` as the JSON parser
gem 'oj'
# gem for dashboard
gem 'rails_admin', '~> 1.0'
# Gem for push notifications
gem 'rpush', :git => "git://github.com/moldedbits/rpush.git", :branch => 'master'
# Rack::Cors provides support for Cross-Origin Resource Sharing (CORS) 
 for Rack compatible web applications.
gem 'rack-cors', :require => 'rack/cors'

# Gem for scheduling tasks
gem 'rufus-scheduler'

gem 'jquery-rails', '~> 4.2', '>= 4.2.1'
gem 'bootstrap-sass', '~> 3.3', '>= 3.3.7'
gem 'sass-rails', '~> 5.0', '>= 5.0.6'
gem 'haml', '~> 4.0', '>= 4.0.7'
gem 'newrelic_rpm'
# Use ActiveModel has_secure_password
gem 'bcrypt', '~> 3.1.7'
gem 'geocoder'
# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development

# Use Rack CORS for handling Cross-Origin Resource Sharing (CORS), making cross-origin AJAX possible
# gem 'rack-cors'

gem 'obscenity', '~> 1.0', '>= 1.0.2'
group :development, :test do
  gem 'dotenv-rails'
  gem 'factory_girl_rails'
  gem 'shoulda-matchers', '~> 3.1', '>= 3.1.1'
  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
  gem 'byebug', platform: :mri
  gem 'rspec-rails', '~> 3.5'
  gem 'airborne'
  gem 'faker', '~> 1.6', '>= 1.6.6'
  # Use sqlite3 as the database for Active Record
end
gem 'rake', '>=11.3.0'
gem 'scout_apm'
group :development do
  gem 'sqlite3'
  gem 'listen', '~> 3.0.5'
  gem 'letter_opener'
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem 'spring'
  gem 'spring-watcher-listen', '~> 2.0.0'
 end
gem 'exception_notification'
group :production do
  gem 'pg'
end
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

I dumped production database and ran it locally with derailed_benchmarks gem. I used this feature to get memory size on runtime.

PATH_TO_HIT=/users/new bundle exec derailed exec perf:mem_over_time

Here is logs of above command

86.640625
41.62109375
36.15625
86.3671875
73.2265625
72.78515625
93.78515625
163.53515625
192.04296875
168.1953125
221.4609375
281.02734375
92.8046875
315.203125
150.9609375
209.9453125
324.18359375
327.5390625
346.625
346.76953125
353.06640625
355.0
360.96484375
369.99609375
385.41015625
388.75
391.203125
378.58203125
375.23046875
387.6640625
389.33984375
390.8828125
399.21484375
415.59375
415.8359375
419.88671875
435.46484375
437.91796875
430.77734375
417.00390625
425.43359375
425.58984375
430.51171875
425.8671875
430.2578125
431.16796875
390.44140625
358.81640625
393.890625
399.44140625
403.44921875
412.859375
363.3515625
359.390625
379.11328125
398.55859375
400.31640625
406.91015625
423.31640625
420.875
425.875
426.91796875
426.5859375
430.390625
386.78515625
377.77734375
374.86328125
252.625

I used tops to see memory size and hit API 50 times locally. Here are logs of top Tops stats

I'm using new relic for both development and production. It shows that major portion of response time is consumed by controller method.

Answer 1

Model.all

will hydrate all your ActiveRecord objects at once. This is a very bad idea. Especially if you are eager_loading three other associated models with includes.

I suggest you use the #find_each method instead to iterate over the collection, allowing the garbage collector to reclaim the memory consumed by each batch as it is processed.

Also returning a single response with this much data is not recommended. A paging system should be implemented on your api which gives the user the opportunity to progressively consume the data.

Answer 2

Instead of returning all the 13k+ items at once you can return it in a batch. That way you won't be consuming as much RAM. I would suggest you to return the records in batches of 1000 using the function "find_each". You can check this link.

Answer 3

You could disable your GC locally to find out what is taking so much memory and try to optimize it.

But here is some tries:

• Are you including correct relations? Apparently, you only need to include user: :opponent.
• class.class_eval seems a really bad thing to be in your each block.