CODEWITHSUNDEEP
pythonweb-scrapingscrapy
user45857
user45857

Reputation: 123

scrape information from multiple urls using a for loop in scrapy

I want to scrape information from multiple urls. I use the following code but it doesn't work. May someone please points me to where I have gone wrong? 

import scrapy

class spider1(scrapy.Spider):
    name = "spider1"
    domain = "http://www.amazon.com/dp/"
    ASIN = ['B01LA6171I', 'B00OUKHTLO','B00B7LUVZK']

    def start_request(self):
        for i in ASIN:
            yield scrapy.Request(url=domain+i,callback = self.parse)

    def parse(self, response):
       title =response.css("span#productTitle::text").extract_first().strip()
       ASIN_ext = response.xpath("//input[@name='ASIN']/@value").extract_first()
       data = {"ASIN":ASIN_ext,"title":title,}
       yield data

Upvotes: 0

Views: 3745

Answers (1)

Joe D
Joe D

Reputation: 388

You just need to add an 's' to the first function

def start_requests(self):

Subtle difference, but Scrapy looks for that specific function so it has to match perfectly.

Upvotes: 6

Related Questions