CODEWITHSUNDEEP
pythonscrapyweb-crawler
dougj
dougj

Reputation: 135

Scrapy only returning first result from the page

I have the following spider, it is simple just trying to parsing the product title, url and price from the single category page. But the problem is spider only getting first result from the page and that is it. I don't understand it can anyone explain this behavior.

url : website to scrape

Spider:

import scrapy
from scrapy.crawler import CrawlerProcess


class VapeSpider(scrapy.Spider):
    name = "vape"

    # custom_settings = {
    #     "FEED_FORMAT": "csv",
    #     "FEED_URI": "vape.csv",
    #     "LOG_FILE": "vape.log",
    # }
    def start_requests(self):
        yield scrapy.Request(
            "https://buyeliquidonline.com.au/product-category/geek-vape/",
            callback=self.parse,
        )

    def parse(self, response):
        for prod in response.css("ul.products:nth-child(2)"):
            yield {
                "title": prod.css("h2.woocommerce-loop-product__title")
                .css("a::text")
                .get()
            }


process = CrawlerProcess()

process.crawl(VapeSpider)

process.start()

Upvotes: 0

Views: 124

Answers (1)

Md. Fazlul Hoque
Md. Fazlul Hoque

Reputation: 16187

The problem was in css element selection. ul.products:nth-child(2) select entire selected page once. You need to select all containers lies on li tag. So you need ul.products:nth-child(2) li then use for loop

import scrapy
from scrapy.crawler import CrawlerProcess


class VapeSpider(scrapy.Spider):
    name = "vap"

    # custom_settings = {
    #     "FEED_FORMAT": "csv",
    #     "FEED_URI": "vape.csv",
    #     "LOG_FILE": "vape.log",
    # }
    def start_requests(self):
        yield scrapy.Request(
            "https://buyeliquidonline.com.au/product-category/geek-vape",
            callback=self.parse,
        )

    def parse(self, response):
        for prod in response.css("ul.products:nth-child(2) li"):
            yield {
                "title": prod.css("h2.woocommerce-loop-product__title").css("a::text").get()
            }


process = CrawlerProcess()

process.crawl(VapeSpider)

process.start()

Output:

{'title': 'Geekvape Aegis Boost Empty Pod Cartridge 3.7ml'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Aegis Boost Pod Kit Luxury Edition 1500mah'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Aegis Hero Pod Kit 1200mah 4ml'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Aegis Hero Replacement Pod Cartridge'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Aegis Legend Kit With Z Sub Ohm Tank 5ml'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Aegis Max Starter Kit'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Aegis Solo 100W Starter Kit'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Aegis X 200w Starter Kit W/ Zeus'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Aero 5ml replacement glass'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Alpha 4ml  Replacement Glass'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Boost Replacement Coils'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Cerberus 5.5ml replacement glass'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape G Coil Zeus Tank'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Super Mesh Coils'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Wenax K1 Pod System'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Zeus replacement glass'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape Zeus sub ohm tank'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Geekvape ZX RTA'}
2022-03-23 06:04:27 [scrapy.core.scraper] DEBUG: Scraped from <200 https://buyeliquidonline.com.au/product-category/geek-vape/>
{'title': 'Wenax replacement pods'}

Upvotes: 1

Related Questions