CODEWITHSUNDEEP
pythonscrapy
patrick.s
patrick.s

Reputation: 43

Scrapy not passing correct item through meta attribute

I'm having a similar problem that the individual below had. I'm trying to pass an item using the meta attribute. I'm seeing the correct number of items outputted, but they are duplicates of a single item. Could someone help? I'm guessing mby the response to the previous individual's post that this should be an obvious fix.

https://github.com/scrapy/scrapy/issues/1257

def parse(self, response):
# some treatment
# a loop
     request = scrapy.Request(url=<calculated_url>, callback=parseChapter)
     request.meta['item'] = # a dictionary containing some data of the just parsed page
     yield request

def parseChapter(self, response):
# some treatment
# a loop
     request = scrapy.Request(url=<calculated_url>, callback=parseCategory)
     request.meta['item'] = # a dictionary containing some data of the just parsed page
     # print request.meta['item'] is good and different in every iteration
     yield request

def parseCategory(self, response):
# print response.meta['item'] is not good because it displays the same value many times
# for every new call of parseChapter, meta['item'] received is always the same
# some treatment

Upvotes: 2

Views: 952

Answers (1)

Frederic Bazin
Frederic Bazin

Reputation: 1529

Most likely, you modifying the item at each iteration of the for loop instead of creating a new one.

As a consequence all request are being sent with the same value. i.e. the last value of the item variable.

def parseChapter(self, response):
# some treatment
# a loop
     request = scrapy.Request(url=<calculated_url>, callback=parseCategory)
     request.meta['item'] = my_item_dict.copy()
     # print request.meta['item'] is good and different in every iteration
     yield request

Upvotes: 2

Related Questions