CODEWITHSUNDEEP
rubyweb-scrapingnokogiriwatir
user3813158
user3813158

Reputation: 11

Structuring Nokogiri output without HTML tags

I got Ruby to travel to a web site, iterate through a list of campaigns and scrape the pages for specific data. The problem I have now is getting it from the structure Nokogiri gives me, and outputting it into a readable form.

campaign_list = Array.new
campaign_list.push(1042360, 1042386, 1042365, 992307)

browser = Watir::Browser.new :chrome
browser.goto '<redacted>'
browser.text_field(:id => 'email').set '<redacted>'
browser.text_field(:id => 'password').set '<redacted>'
browser.send_keys :enter

file = File.new('hourlysales.csv', 'w')
data = {}

campaign_list.each do |campaign|
  browser.goto "<redacted>"

  if browser.text.include? "Application Error"
    puts "Error loading page, I recommend restarting script"
    # Possibly automatic restart of script
  else
    hourly_data = Nokogiri::HTML.parse(browser.html).text   
    # file.write data
    puts hourly_data
  end

This is the output I get:

{"views":[[17,145],[18,165],[19,99],[20,71],[21,31],[22,26],[23,10],[0,15],[1,1],      [2,18],[3,19],[4,35],[5,47],[6,44],[7,67],[8,179],[9,141],[10,112],[11,95],[12,46],[13,82],[14,79],[15,70],[16,103]],"orders":[[17,10],[18,9],[19,5],[20,1],[21,1],[22,0],[23,0],[0,1],[1,0],[2,1],[3,0],[4,1],[5,2],[6,1],[7,5],[8,11],[9,6],[10,5],[11,3],[12,1],[13,2],[14,4],[15,6],[16,7]],"conversion_rates":[0.06870229007633588,0.05442176870748299,0.050505050505050504,0.014084507042253521,0.03225806451612903,0.0,0.0,0.06666666666666667,0.0,0.05555555555555555,0.0,0.02857142857142857,0.0425531914893617,0.022727272727272728,0.07462686567164178,0.06134969325153374,0.0425531914893617,0.044642857142857144,0.031578947368421054,0.021739130434782608,0.024390243902439025,0.05063291139240506,0.08571428571428572,0.06741573033707865]}

The arrays stand for { views [[hour, # of views], [hour, # of views], etc. }. Same with orders. I don't need conversion rates.

I also need to add the values up for each key, so after doing this for 5 pages, I have one key for each hour of the day, and the total number of views for that hour. I tried a couple each loops, but couldn't make any progress.

I appreciate any help you guys can give me.

Upvotes: 1

Views: 147

Answers (1)

Jordan Running
Jordan Running

Reputation: 106147

It looks like the output (which from your code I assume is the content of hourly_data) is JSON. In that case, it's easy to parse and add up the numbers. Something like this:

require "json" # at the top of your script
# ...

def sum_hours_values(data, hours_values=nil)
  # Start with an empty hash that automatically initializes missing keys to `0`
  hours_values ||= Hash.new {|hsh,hour| hsh[hour] = 0 }

  # Iterate through the [hour, value] arrays, adding `value` to the running
  # count for that `hour`, and return `hours_values`
  data.each_with_object(hours_values) do |(hour, value), hsh|
    hsh[hour] += value
  end
end

# ... Watir/Nokogiri stuff here...

# Initialize these so they persist outside the loop
hours_views, orders_views = nil

campaign_list.each do |campaign|
  browser.goto "<redacted>"

  if browser.text.include? "Application Error"
    # ...
  else
    # ...

    hourly_data_parsed = JSON.parse(hourly_data)

    hours_views = sum_hours_values(hourly_data_parsed["views"], hours_views)
    hours_orders = sum_hours_values(hourly_data_parsed["orders"], orders_views)
  end
end

puts "Views by hour:"
puts hours_views.sort.map {|hour_views| "%2i\t%4i" % hour_views }

puts "Orders by hour:"
puts hours_orders.sort.map {|hour_orders| "%2i\t%4i" % hour_orders }

P.S. There's a really nice recursive version of sum_hours_values I didn't include since the iterative version is clearer to most Ruby programmers. If you're into recursion I leave it as an exercise for you. ;)

Upvotes: 1

Related Questions