CODEWITHSUNDEEP
rubycocoamemory-managementmemory-leaksrubymotion
MichaelC
MichaelC

Reputation: 357

How to prevent memory leaks in RubyMotion when reading large files in loops

RubyMotion is supposed to do automatic memory management:

RubyMotion provides automatic memory management; you do not need to reclaim unused objects.

but, when reading large files in a loop, I encounter huge memory leaks: hundreds of MB/s per seconds, exactly as if my reading buffer was never released.

The leaks mostly go away if I use release on the reading buffer on every loop. The problem is, release makes the application crash when the loop is finished.

  def readBigBinaryFile(file)
#   PURE RUBY WOULD BE
#   source=File.open(file,'r')
    source =NSFileHandle.fileHandleForReadingAtPath(file)
    buffer_size=4096
    offset=0
    size=File.size(file)
    while ( offset + buffer_size ) <= size
#      PURE RUBY WOULD BE
#      source.seek(offset) 
#      abuffer = source.read( buffer_size )
#      abuffer=abuffer.to_s

      source.seekToFileOffset(offset)
      abuffer = source.readDataOfLength(buffer_size)
      offset+=buffer_size
      @dataPointer ||= Pointer.new(:uchar,4) 
      abuffer.getBytes(@dataPointer, length: 4)
      # memory leaks very rapidly in GBs if we don't release the buffer…
      # but this relase action will make the application crash once the doSomething lookp is finished
      abuffer.release
    end
    source.closeFile
    return false
  end

The loop is:

x=0
while x < 10000
  my_scan_binary_instance=Scan_binary.new()        result=my_scan_binary_instance.readBigBinaryFile(NSBundle.mainBundle.pathForResource("sample1MBfile", ofType:"img"))
  puts result.to_s
  x+=1
end
puts "if we have used 'abuffer.release', we are going to crash now"

I tested a pure-Ruby implementation, and had no memory leak at all, and no need for the release call.

I found "How do I prevent memory leak when I load large pickle files in a for loop?" about a memory leak in a Python loop, but the accepted solution doing abuffer=nil at the beginning of the while block in readBigBinaryFile did not work.

Is this a bug in RubyMotion's automatic memory management, or is this expected? And most importantly, how can I read big files in loops without increasing the memory usage of my RubyMotion app?

I have created a gist with the working pure Ruby implementation, and a repo of a sample application reproducing the crash.

Upvotes: 0

Views: 264

Answers (2)

vacawama
vacawama

Reputation: 154631

Try wrapping your loop body in autorelease_pool do ... end. This should cause the autoreleased objects to be freed each loop. Assigning nil to abuffer at the end of the loop will allow the buffer memory to be freed as there will be no more references to it.

while ( offset + buffer_size ) <= size
  autorelease_pool do
    source.seekToFileOffset(offset)
    abuffer = source.readDataOfLength(buffer_size)
    offset+=buffer_size
    @dataPointer ||= Pointer.new(:uchar,4) 
    abuffer.getBytes(@dataPointer, length: 4)

    abuffer = nil
  end
end

Upvotes: 2

Eric Henderson
Eric Henderson

Reputation: 26

I don't know if this will fix it for sure, but maybe try replacing

abuffer = source.readDataOfLength(buffer_size)

with

abuffer = WeakRef.new(source.readDataOfLength(buffer_size))

See "4.1. Weak References" for info on WeakRef.

Making this change would mean you would also remove the call to abuffer.release.

Upvotes: 1

Related Questions