CODEWITHSUNDEEP
ruby-on-railsruby-on-rails-4herokumemory-leaksbackground-process
Meltemi
Meltemi

Reputation: 38359

Troubleshooting Heroku R14 & R15 memory errors on background process

I'm looking for help troubleshooting some R14 and R15 memory errors I'm finding fairly often in the logs of a Rails app running on Heroku.

Jan 15 08:34:12 myapp-production app/worker.1:  2018-1-15T16:34:11+0000: [Worker(host:a833f658-b5 pid:4)] Job StoryAssetMailer#perform (id=7776) RUNNING 
Jan 15 08:36:31 myapp-production heroku/worker.1: Process running mem=518M(100.0%) 
Jan 15 08:36:31 myapp-production heroku/worker.1: Error R14 (Memory quota exceeded) 
Jan 15 08:36:33 myapp-production heroku/worker.1: Error R15 (Memory quota vastly exceeded)

I've set WEB_CONCURRENCY=1 as recommended on Heroku's R14 support page. It made little difference.

Problem is I don't really know what tool(s) to use to measure where the memory spike (leak?) is occurring? I can find the offending method (included below) but looking for ideas in how to narrow down exactly where the problem & then I'll worry about how to fix it.

Background process creating errors is Job StoryAssetMailer#perform:

# app/jobs/story_asset_mailer.rb
require Rails.root.join('lib', 's3_store')
require 'zip'

class StoryAssetMailer
  def initialize(recipient:, story:)
    @recipient = recipient
    @story = story
  end

  def perform
    stored_file = S3Store.new(
      zip_file_stream,
      @story.download_safe_title(
        extension: 'zip'
      )
    ).store(method: :sysread)

    Mailer.story_images(
      recipient: @recipient,
      story: @story,
      file_url: stored_file.url
    ).deliver
  end

  def zip_file_stream
    manifest = ZipManifest.build(media_files_for_story)
    build_zip_file(manifest)
  end   
  ...
end

Related model:

#app/models/zip_manifest.rb
require 'open-uri'

class ZipManifest
  def initialize(manifest: nil)
    @manifest ||= manifest
  end

  def each_file
    @manifest.each do |contents, filename|
      yield filename, contents
    end
  end

  def self.build(images)
    new(
      manifest:
        images.map do |image|
          [open(image.url), image.filename]
        end
    )
  end
end

Edit: Is this thing on?

Upvotes: 5

Views: 9631

Answers (2)

Stan Mazhara
Stan Mazhara

Reputation: 826

You could use https://devcenter.heroku.com/articles/log-runtime-metrics heroku plugin to watch your memory consumption in real time.

Your process memory usage directly depends on the size of files your are processing and you're using the smallest dyno (512Mb). You may want to scale it up.

Also, WEB_CONCURRENCY is generally used to control web dynos concurrency, not workers.

R15 error (quota vastly exceeded) means you are over 200% (https://devcenter.heroku.com/articles/error-codes#r15-memory-quota-vastly-exceeded) which causes heroku to kills the process.

R14 error (memory quota exceeded) is a warning when you are over 100% but below 200% of memory quota. It only affects performance (the dyno starts swapping).

So, put a max asset file size limit in your app and make sure that asset file size + your app slug size + some safety margin is within your dyno memory limits.

Upvotes: 3

Will Nguyen
Will Nguyen

Reputation: 348

Please double-check your file. Because RAM will allocate memory for your file as well. If your file is 500M so that total memory should be (500M + memory for background processes).

Upvotes: 4

Related Questions