CODEWITHSUNDEEP
pythonweb-scrapingscrapy
NumenorForLife
NumenorForLife

Reputation: 1746

Scrapy Not Returning After Yielding a Request

Similar to the person here: Scrapy Not Returning Additonal Info from Scraped Link in Item via Request Callback, I am having difficulty accessing the list of items I build in my callback function. I have tried building the list both in the parse function (but doesn't work because the callback hasn't returned), and the callback, but neither have worked for me.

I am trying to return all the items that I build from these requests. Where do I call "return items" such that the item has been fully processed? I am trying to replicate the tutorial (http://doc.scrapy.org/en/latest/intro/tutorial.html#using-our-item) Thanks!!

The relevant code is below:

class ASpider(Spider):
    items = []
...
  def parse(self, response):
      input_file = csv.DictReader(open("AFile.txt"))
      x = 0
      for row in input_file:
          yield  Request("ARequest", 
          cookies = {"arg1":"1", "arg2":row["arg2"], "style":"default", "arg3":row["arg3"]}, callback = self.aftersubmit, dont_filter = True)

  def aftersubmit(self, response):
    hxs = Selector(response)
    # Create new object..
    item = AnItem()
    item['Name'] = "jsc"
    return item

Upvotes: 1

Views: 1183

Answers (1)

alecxe
alecxe

Reputation: 473763

You need to return (or yield) an item from the aftersubmitcallback method. Quote from docs:

In the callback function, you parse the response (web page) and return either Item objects, Request objects, or an iterable of both.

def aftersubmit(self, response):
    hxs = Selector(response)

    item = AnItem()
    item['Name'] = "jsc"
    return item

Note that this particular Item instance doesn't make sense since you haven't really put anything from the response into it's fields.

Upvotes: 1

Related Questions