CODEWITHSUNDEEP
rubycsvbufferlarge-files
eee
eee

Reputation: 280

Does Ruby's CSV.open buffer to memory and write all at once?

Will CSV.open store data in memory and write to file one time when the block exits, or it will automatically write in many batches?

require 'csv'

CSV.open('result.csv', 'wb') do |csv|
  while row = next_row
    csv << row
  end
end

Upvotes: 11

Views: 4393

Answers (1)

user513951
user513951

Reputation: 13612

CSV.open will write to the underlying OS when the block closes, and it will also write every time the buffer fills and is flushed, which will happen automatically. (In my Ruby install, it happens at 8196 bytes.) You can also add csv.flush to your block to force it to write sequentially.

require 'csv'

CSV.open('result.csv', 'wb') do |csv|
  while row = next_row
    csv << row  # Without flush, writes to underlying OS only when buffer fills
    csv.flush   # Adding this forces a write to underlying OS
  end
end             # Exiting block writes to underlying OS and closes file handle

(Note that the writes are to the underlying OS, which may buffer the stream again before actually writing to disk.)

Upvotes: 16

Related Questions