CODEWITHSUNDEEP
ruby
Rajagopalan
Rajagopalan

Reputation: 6064

When array count increases, getting this error "failed to allocate memory (NoMemoryError) zlib(finalizer): the stream was freed prematurely."

I am having this weird error

failed to allocate memory (NoMemoryError)
zlib(finalizer): the stream was freed prematurely.

When I try to do this, it goes well with the first iteration

array.each_slice(40000) do |result|
   #writing this result array into the excel
end

But when the second iteration goes, it thows above mentioned error. What's the problem here? Someone can help me here?

The same problem is occurring in the first iteration if the count is 50000.

Upvotes: 1

Views: 296

Answers (1)

Vamsi Pavan Mahesh
Vamsi Pavan Mahesh

Reputation: 250

Well, you are probably running out of memory as the error says.

Maybe try iterating in 1000 batches over 10_000

You are better off creating a file and writing to it rather than creating a giant in memory Array. Note that writing to a file is optimal only when your hardware supports, you might/might not have the same hardware as your MacBook on your production server. So you might want to check with that.

Also, It's not a bad idea to call GC.start if this is a background job for every 5000 records or so.

Finally, check the memory allocations using something like memory profiler

Upvotes: 1

Related Questions