CODEWITHSUNDEEP
rubynokogirirspec2
wyc
wyc

Reputation: 55343

Nokogiri scraping test failing?

I have the following code:

  url       = "http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dpets&field-keywords="
  data      = Nokogiri::HTML(open(url))

  department   = data.css('#ref_2619534011')

  @department_hash = {}
  department.css('li').drop(1).each do | department |
    department_title = department.css('.refinementLink').text
    department_count = department.css('.narrowValue').text[/[\d,]+/].delete(",").to_i
    @department_hash[:department] ||= {}
    @department_hash[:department]["Pet Supplies"] ||= {}
    @department_hash[:department]["Pet Supplies"][department_title] = department_count
  end

So when I do this <%= @department_hash %> in a template I get this: 

{:department=>{"Pet Supplies"=>{"Birds"=>15918, "Cats"=>245418, "Dogs"=>513869, "Fish & Aquatic Pets"=>47182, "Horses"=>14774, "Insects"=>358, "Reptiles & Amphibians"=>5834, "Small Animals"=>19806}}}

I created a spec for app.rb (I'm using Sinatra):

app_spec.rb:

require File.dirname(__FILE__) + '/app.rb'

describe "Department" do
  it "should scrap correct string" do
    expect(@department_hash).to eq '{:department=>{"Pet Supplies"=>{"Birds"=>15918, "Cats"=>245418, "Dogs"=>513869, "Fish & Aquatic Pets"=>47182, "Horses"=>14774, "Insects"=>358, "Reptiles & Amphibians"=>5834, "Small Animals"=>19806}}}' 
  end
end

But the test fails:

  1) Department should scrap correct string
     Failure/Error: expect(@department_hash).to eq '{:department=>{"Pet Supplies"=>{"Birds"=>15918, "Cats"=>245418, "Dogs"=>513869, "Fish & Aquatic Pets"=>47182, "Horses"=>14774, "Insects"=>358, "Reptiles & Amphibians"=>5834, "Small Animals"=>19806}}}'

       expected: "{:department=>{\"Pet Supplies\"=>{\"Birds\"=>15918, \"Cats\"=>245418, \"Dogs\"=>513869, \"Fish & Aquatic Pets\"=>47182, \"Horses\"=>14774, \"Insects\"=>358, \"Reptiles & Amphibians\"=>5834, \"Small Animals\"=>19806}}}"
            got: nil

       (compared using ==)
     # ./app_spec.rb:5:in `block (2 levels) in <top (required)>'

EDIT:

I tried this:

expect(@department_hash[:department]["Pet Supplies"].keys).to eq '["Birds", "Cats", "Dogs", "Fish & Aquatic Pets", "Horses", "Insects", "Reptiles & Amphibians", "Small Animals"]'

But test fails also:

2) Department should scrap correct keys Failure/Error: expect(@department_hash[:department]["Pet Supplies"].keys).to eq '["Birds", "Cats", "Dogs", "Fish & Aquatic Pets", "Horses", "Insects", "Reptiles & Amphibians", "Small Animals"]' NoMethodError: undefined method []' for nil:NilClass # ./app_spec.rb:9:inblock (2 levels) in '

What could be the reason?

Upvotes: 0

Views: 186

Answers (1)

Justin Ko
Justin Ko

Reputation: 46846

@department_hash is not defined within the scope of the test.

If you take a simple example:

require 'rspec/autorun'

@department_hash = 2
puts defined?(@department_hash)
#=> "instance-variable"

describe "Department" do
  it "should scrap correct string" do
        puts defined?(@department_hash)
        #=> "" (ie not defined)
    end
end

You can see that @department_hash is defined in the main, but it is not defined within the test.

You need to run your app code from within the scope of the test. For example, moving the code into the test, @department_hash will no longer be nil. 

require 'rspec/autorun'
require 'nokogiri'
require 'open-uri'

describe "Department" do
  it "should scrap correct string" do
    url       = "http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dpets&field-keywords="
    data      = Nokogiri::HTML(open(url))

    department   = data.css('#ref_2619534011')

    @department_hash = {}
    department.css('li').drop(1).each do | department |
      department_title = department.css('.refinementLink').text
      department_count = department.css('.narrowValue').text[/[\d,]+/].delete(",").to_i
      @department_hash[:department] ||= {}
      @department_hash[:department]["Pet Supplies"] ||= {}
      @department_hash[:department]["Pet Supplies"][department_title] = department_count
    end       

    expect(@department_hash).to eq({:department=>{"Pet Supplies"=>{"Birds"=>17556, "Cats"=>245692, "Dogs"=>516246, "Fish & Aquatic Pets"=>47424, "Horses"=>15062, "Insects"=>358, "Reptiles & Amphibians"=>5835, "Small Animals"=>19836}}})
  end
end

Note that your test should be eq(hash) rather than eq 'hash' (ie you want to compare hashes rather than a string to a hash.

Update - Code Extracted to a Class:

Moving the app code into the test is not ideal if it is meant to be reusable else where. Instead, it might be better to create a method or class for your application code and then call that code from the test.

require 'rspec/autorun'
require 'nokogiri'
require 'open-uri'

# This class could be placed in your app.rb  
class DepartmentScraper
  def scrape_page()
    url       = "http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dpets&field-keywords="
    data      = Nokogiri::HTML(open(url))

    department   = data.css('#ref_2619534011')

    @department_hash = {}
    department.css('li').drop(1).each do | department |
      department_title = department.css('.refinementLink').text
      department_count = department.css('.narrowValue').text[/[\d,]+/].delete(",").to_i
      @department_hash[:department] ||= {}
      @department_hash[:department]["Pet Supplies"] ||= {}
      @department_hash[:department]["Pet Supplies"][department_title] = department_count        
    end

    return @department_hash
  end
end

describe "Department" do
  it "should scrap correct string" do
    department_hash = DepartmentScraper.new.scrape_page()

    expect(department_hash).to eq({:department=>{"Pet Supplies"=>{"Birds"=>17556, "Cats"=>245692, "Dogs"=>516246, "Fish & Aquatic Pets"=>47424, "Horses"=>15062, "Insects"=>358, "Reptiles & Amphibians"=>5835, "Small Animals"=>19836}}})
  end
end

Upvotes: 1

Related Questions