CODEWITHSUNDEEP
pythonxpathscrapyweb-crawler
Lucas Guidi
Lucas Guidi

Reputation: 65

Scrapy collecting information only from the first item on each page, why?

I have the following spider, but it only collects the first items on each page.

Can someone explain to me why? I couldn't find my mistake.

import scrapy

class PerfumesSpider(scrapy.Spider):
    name = 'perfumes'
    allowed_domains = ['www.fragrancenet.com']
    start_urls = ['https://www.fragrancenet.com/fragrances']
    

    def parse(self, response):
        for perfumes in response.xpath("//div[@id='resultSet']"):
            
            #nome = perfumes.xpath(".//span[@class='brand-name']/text()").get(),
            link = perfumes.xpath(".//p[@class='desc']/a/@href").get()
        
            yield response.follow(url=link, callback=self.parse_produto, meta={'link' : link})
 
        next_page = response.xpath("//a[@data-rel='next']/@href").get()
 
        if next_page:
            yield scrapy.Request(url=next_page, callback=self.parse)

    def parse_produto(self, response):
        link = response.request.meta['link']
        for produto in response.xpath("//div[@class='topZone cf']"):

            yield{
            'link': link,
            'gender': produto.xpath(".//span[@class='genderBar desktop']/span/span/text()").get(),
            'name': produto.xpath(".//span[@class='productTitle']/text()").get(),            
            'year': produto.xpath(".//ul[@class='notes cf']/li[3]/span[2]/text()").get(),            
            'brand': produto.xpath("normalize-space(.//p[@class='uDesigner']/a/text())").get(),
            'size': produto.xpath(".//span[@class='sr-only']/text()").getall(),
            'price': produto.xpath(".//div[@class='pricing']/text()").getall(),
            'discount': produto.xpath(".//div[@class='fnet-offer']/a/span/span/text()").get(), 
            }

If anyone can help me, I will be grateful

Upvotes: 0

Views: 49

Answers (1)

Felix Eklöf
Felix Eklöf

Reputation: 3720

You need to select the div's inside resultSet, try changing from this:

for perfumes in response.xpath("//div[@id='resultSet']"):

to something like this (im not sure of the xpath code, youll have to double check that):

for perfumes in response.xpath("//div[@id='resultSet']//div[@class='resultItem']"):

Upvotes: 1

Related Questions