CODEWITHSUNDEEP
pythonscrapy
merlin
merlin

Reputation: 2897

How to pass variable to parse_item method in Scrapy?

I am trying to pass a variable (sku) to the parse_item method within Scrapy. This results in:

TypeError: init() got an unexpected keyword argument 'sku'

My code:

def start_requests(self):
    for sku in self.SKUs:
        url = 'https://www.test' + sku
        if validators.url(url):
            yield scrapy.Request(
                url=url,
                sku=sku,
                callback=self.parse_item,
            )
        else:
            print("Invalid URL ", format(url))

def parse_item(self, response):

    shop['sku'] = self.sku

How can the variable sku be passed to the parse_item method?

Upvotes: 1

Views: 52

Answers (1)

gangabass
gangabass

Reputation: 10666

You need to use cb_kwargs param (or meta in old Scrapy versions):

def start_requests(self):
    for sku in self.SKUs:
        url = 'https://www.test' + sku
        if validators.url(url):
            yield scrapy.Request(
                url=url,
                cb_kwargs={
                    'sku': sku,
                },
                callback=self.parse_item,
            )
        else:
            print("Invalid URL ", format(url))

def parse_item(self, response, sku):
    shop['sku'] = sku

Upvotes: 1

Related Questions