CODEWITHSUNDEEP
rubyfile
T0ny lombardi
T0ny lombardi

Reputation: 1920

Fastest Way to write large files

I have a script thats writes a series of numbers on a line millions of times. What's the fastest way for ruby to write to a file like that? Right now I'm calculating that it will take me about 21 hours to write 258 million lines to this file. It's also going to be roughly 8-15gigs. any suggestions ?

def log_file(text) 
  File.open(File.join("combo_numbers.txt"), 'a+') do |f|
    f.write("#{number}\n") 
  end
end


250000000.times do
  # math_answer =[]
  # math_answer = math 
  log_file(math_answer)
end

Upvotes: 1

Views: 1661

Answers (1)

maček
maček

Reputation: 77778

You could write a small logging class that only opens the file once instead of reopening it for each write

class Logger

  def initialize path, mode = "w"
    @path = path
    @mode = mode
  end

  def write data
    handle.write data + "\n"
  end

  private

  def handle
    @f ||= File.open(@path, @mode)
  end

  def close
    @f && @f.close
  end

end

Usage

logger = Logger.new "combo_numbers.txt"

250_000_000.times do
  logger.write "foo"
end

logger.close

Upvotes: 1

Related Questions