CODEWITHSUNDEEP
pythonscrapy
user6669314
user6669314

Reputation: 31

Passing xPath as argument to Scrapy

I'm trying to write a generic crawler for a single webpage which is called with the following arguments:

The URL and allowed domains arguments seem to be working correctly but I can't get the xPath argument to work.

I'm guessing I need to declare a variable to hold it correct as the other two arguments are assigned to existing class elements.

Here is my spider:

import scrapy
from Spotlite.items import SpotliteItem

class GenericSpider(scrapy.Spider):
   name = "generic"

   def __init__(self, start_url=None, allowed_domains=None, xpath_string=None, *args, **kwargs):
      super(GenericSpider, self).__init__(*args, **kwargs)
      self.start_urls = ['%s' % start_url]
      self.allowed_domains = ['%s' % allowed_domains]
      xpath_string = ['%s' % xpath_string]

   def parse(self, response):
      self.logger.info('Hi, this is an item page! %s', response.url)
      item = SpotliteItem()
      item['url'] = response.url
      item['price'] = response.xpath(xpath_string).extract()
      return item

I get the following error:

Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/home/ubuntu/spotlite/spotlite/spiders/generic.py", line 23, in parse
     item['price'] = response.xpath(xpath_string).extract()

NameError: global name 'xpath_string' is not defined

Any assistance will be appreciated!

Thanks,

Michael

Upvotes: 1

Views: 311

Answers (2)

user6669314
user6669314

Reputation: 31

Adding the variable to initial class declaration fixed the problem.

import scrapy
from spotlite.items import SpotliteItem


class GenericSpider(scrapy.Spider):
   name = "generic"
   xpath_string = ""

   def __init__(self, start_url, allowed_domains, xpath_string, *args, **kwargs):
       super(GenericSpider, self).__init__(*args, **kwargs)
       self.start_urls = ['%s' % start_url]
       self.allowed_domains = ['%s' % allowed_domains]
       self.xpath_string = xpath_string

    def parse(self, response):
       self.logger.info('URL is %s', response.url)
       self.logger.info('xPath is %s', self.xpath_string)
       item = SpotliteItem()
       item['url'] = response.url
       item['price'] = response.xpath(self.xpath_string).extract()
       return item

Upvotes: 0

alecxe
alecxe

Reputation: 473893

Have xpath_string as an instance variable instead:

import scrapy
from Spotlite.items import SpotliteItem

class GenericSpider(scrapy.Spider):
   name = "generic"

   def __init__(self, start_url=None, allowed_domains=None, xpath_string=None, *args, **kwargs):
      super(GenericSpider, self).__init__(*args, **kwargs)
      self.start_urls = ['%s' % start_url]
      self.allowed_domains = ['%s' % allowed_domains]
      self.xpath_string = xpath_string

   def parse(self, response):
      self.logger.info('Hi, this is an item page! %s', response.url)
      item = SpotliteItem()
      item['url'] = response.url
      item['price'] = response.xpath(self.xpath_string).extract()
      return item

Upvotes: 1

Related Questions