CODEWITHSUNDEEP
ruby-on-railsrubycsv
analyticsPierce
analyticsPierce

Reputation: 3025

How to continue processing a block in Ruby after an exception?

I am trying to process some very large tab-separated files. The process is:

  begin
   Dir["#{@data_path}*.tsv"].each do |file|
       begin              
          CSV.foreach(file, :col_sep => "\t") do |row|

           # assign columns to model and save

           end
           @log.info("Loaded all files into MySQL database illu.datafeeds")
       rescue Exception => e
             @log.warn("Unable to process the data feed: #{file} because #{e.message}")
             next
       end
   end

However, when I execute this I get the following error:

Unable to process the file: /Users/XXXXX_2013-06-12.tsv because Illegal quoting in line 153.

The files are too big for me to go in and fix the error rows. I would like the process to continue the loop and process the file even if there are error rows.

Any suggestions?

Thanks.

Upvotes: 2

Views: 2050

Answers (1)

okliv
okliv

Reputation: 3959

just ... rescue nil the row causing the error

you can even log it with logger

before the loop:

error_log ||= Logger.new("#{Rails.root}/log/my.log")

inside the loop instead of just rescue nil use

rescue error_log.info(row.to_s)

in case you get the error before file begins to parse (before .foreach procedure) you can open it as raw file and read it as CSV later - inside the loop (like mentioned here)

..or just rescue full file parsing procedure

 CSV.foreach(file, :col_sep => "\t") do |row|
    ...
 end rescue error_log.info(row.to_s)

Upvotes: 4

Related Questions