CODEWITHSUNDEEP
rubytext-processing
localshred
localshred

Reputation: 2233

How to efficiently parse large text files in Ruby

I'm writing an import script that processes a file that has potentially hundreds of thousands of lines (log file). Using a very simple approach (below) took enough time and memory that I felt like it would take out my MBP at any moment, so I killed the process.

#...
File.open(file, 'r') do |f|
  f.each_line do |line|
    # do stuff here to line
  end
end

This file in particular has 642,868 lines:

$ wc -l nginx.log                                                                                                                                        /code/src/myimport
  642868 ../nginx.log

Does anyone know of a more efficient (memory/cpu) way to process each line in this file?

UPDATE

The code inside of the f.each_line from above is simply matching a regex against the line. If the match fails, I add the line to a @skipped array. If it passes, I format the matches into a hash (keyed by the "fields" of the match) and append it to a @results array.

# regex built in `def initialize` (not on each line iteration)
@regex = /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - (.{0})- \[([^\]]+?)\] "(GET|POST|PUT|DELETE) ([^\s]+?) (HTTP\/1\.1)" (\d+) (\d+) "-" "(.*)"/

#... loop lines
match = line.match(@regex)
if match.nil?
  @skipped << line
else
  @results << convert_to_hash(match)
end

I'm completely open to this being an inefficient process. I could make the code inside of convert_to_hash use a precomputed lambda instead of figuring out the computation each time. I guess I just assumed it was the line iteration itself that was the problem, not the per-line code.

Upvotes: 6

Views: 12752

Answers (3)

hukl
hukl

Reputation: 41

This blogpost includes several approaches to parsing large log files. Maybe thats an inspiration. Also have a look at the file-tail gem

Upvotes: 4

Mike Bethany
Mike Bethany

Reputation:

I just did a test on a 600,000 line file and it iterated over the file in less than half a second. I'm guessing the slowness is not in the file looping but the line parsing. Can you paste your parse code also?

Upvotes: 5

Andrew Amis
Andrew Amis

Reputation: 142

If you are using bash (or similar) you might be able to optimize like this:

In input.rb:

 while x = gets
      # Parse
 end

then in bash:

 cat nginx.log | ruby -n input.rb

The -n flag tells ruby to assume 'while gets(); ... end' loop around your script, which might cause it to do something special to optimize.

You might also want to look into a prewritten solution to the problem, as that will be faster.

Upvotes: 1

Related Questions