CODEWITHSUNDEEP
rubyarraysout-of-memoryruby-2.1
user1032752
user1032752

Reputation: 881

Ruby combining two arrays causes process to run out of memory

Combining two arrays of hashes that have 100k items each causes a process running on a 2GB VM to run out of memory. I'm failing to understand how/why.

Let's say I have a Hash like this and I populate it with 50 key/value pairs.

h = {}
1..50.times{ h[SecureRandom.hex] = SecureRandom.hex}

I place 100k hs into two Arrays as so:

a = []
a1 = []
1..100_000.times{ a << h }
1..100_000.times{ a1 << h }

When I try to add a1 into a, IRB runs out of memory:

2.1.1 :008 > a << a1
NoMemoryError: failed to allocate memory

Are the two arrays really too big to combine in memory? What is the preferred way to accomplish this?

I'm running ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-linux] and the VM has no other processes running on it.

Upvotes: 1

Views: 266

Answers (1)

Ajedi32
Ajedi32

Reputation: 48428

The problem is most likely not that Ruby is running out of memory while performing this operation (especially since there's only one copy of the h hash), but rather that IRB is running out of memory while trying to display the result. Try adding a ; nil after the final line in IRB; that should solve the problem since it will prevent IRB from trying to display the result hash.

Example:

require 'securerandom'
require 'objspace'

h = {}
1..50.times{ h[SecureRandom.hex] = SecureRandom.hex}
a = []
a1 = []
1..100_000.times{ a << h }
1..100_000.times{ a1 << h }
a << a1; nil # Semicolon and nil are for IRB, not needed with regular Ruby

puts "Total memory usage: #{ObjectSpace.memsize_of_all/1000.0} KB"

I get a result of Total memory usage: 7526.543 KB; no where close to 2 GB.

Upvotes: 6

Related Questions