Rails paperclip upload images folder - Errno::EMFILE

Question

First i wanna apologize for my poor english.

I have a piece of code which uploads to server folder of files using gem PaperClip 3.3.1. Which small number of photos (up to 30) every thing looks great until i try upload folder with 50 photos inside. Then i get following error message:

Errno::EMFILE (Too many open files - identify -format %m '/tmp/4209520130906-10816-1kk0w0o.jpg[0]'):

I check ulimit -a in my system and in my enviroment i can have 1200 open files in same time. So then i runned commad:

watch -n1 'lsof -a -p <rails server pid> | wc -l'

And with each iteration number of opened files was increasing

110 - starting count
160 - after first iteration (number from folder)
240 - second iteration
320
376
457
...
1118 - of about 32 iteration
crash ~ :)

I figured out that problem is caused by proc which adds image_path to database and resize it if special format required (during html input parsing)

So in debugger i runned following commad to check what files ruby holds.

ObjectSpace.each_object(File){ |f| puts f.path }

I found that there was 50 files uploaded with name including 'RackMultipart' and after iteration number of files in tmp folder was credited by new files with strange names (probably paperclip temp files). I thought that peperclip or some of app code doesn't close file so in end of proc i added following code:

        now = Time.now.to_formatted_s( :number )[0..7]
        ObjectSpace.each_object(File).map do |f|
          if f.path.include?( "/tmp/" ) && !f.closed?
            unless f.path.include? "RackMultipart#{now}"
              f.close
              f = nil
            end
          end
        end

After that nothing really happened. In debugger i runned command

ObjectSpace.each_object(File){ |f| puts ( f.closed? '+' : '-' ) }

And number of opened files (-) was correct (~160) but command still listed files closed. Shouldn't they disappear after i closed them?

truncated method looks like bellow:

def function(g, files)

    ...

    asset = proc do |img|
      ...
      file = files.find{|f| f.original_filename == src}
      if file

        geo = Paperclip::Geometry.from_file file.path

        if(geo.width <= 20 && geo.height <= 20)
          ...
        else
          asset = figure.assets.build
          asset.file = file ## HERE RUNS
          ...
        end

        now = Time.now.to_formatted_s( :number )[0..7]
        ObjectSpace.each_object(File).map do |f|
          if f.path.include?( "/tmp/" ) && !f.closed?
            unless f.path.include? "RackMultipart#{now}"
              f.close
              f = nil
            end
          end
        end
      else
        ...
      end
    end

    ...

      while img = g.find_first(".//img")
        asset.call(img) # !HERE HAPPENS
        img.remove!
      end

 end # end function

Asset model definition:

class Asset < ActiveRecord::Base
  ...

  has_attached_file :file, url: "/system/:class/:attachment/:id/:style_:filename",
                    styles: lambda { |attachment| attachment.instance.switch_styles },
                    :convert_options => {medium: lambda{|asset| asset.is_table? ? "-units PixelsPerInch -resample 120 -strip" : "-strip"}, 
                                            all: '-strip'}




  validates_attachment :file, content_type: { content_type: ["image/jpg","image/png","image/gif","image/jpeg","image/tiff"] },
                       size: { in: 0..10.megabytes }

  def is_table?
    ...
  end

  def switch_styles
    self.file.content_type == "image/tiff" ?
    { backup: "100%", original: ["100%", :png], medium: [self.is_table? ? "" : "800x600>", :png], small: ["300x300>", :png], formula: ['50%', :png] } :
    { medium: self.is_table? ? "" : "800x600>", small: "300x300>", formula: '50%' }
  end

end

I hope you'll understand what i wrote ;)

Thanks in advance 4 your help.

Answer 1

Ok. That was nothing wrong with this code and everything works fine. But someone sometime disabled GarbageColector in parent parent class. Probably to speed up parsing process. So now when number of files exceeded 70% of ulimit i turn on GC and if it clean unnecesary temp files i turn off it again.

---

Edit: Function to run and then disable again GC

  def gc_check
    if ObjectSpace.each_object(File).to_a.size > 850  
      GC.start if GC.enable
    else
      GC.disable
    end
  end

hope this helps you.