CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Diego Pesco Alcalde
Diego Pesco Alcalde

Reputation: 35

Scrapy works in shell but not in the code

I am facing an issue while developing my first spider in scrapy. I am able to get the proper information in scrapy shell but it does not work when I implement it in the code. I've read similar posts here but I still was not able to figure out what I'm doing wrong.

import scrapy
from scrapy.loader import ItemLoader
from ..items import ScrapingamazonItem

class AmazonSpiderSpider(scrapy.Spider):
    name = 'amazon_spider'
    start_urls = ['https://www.amazon.com/s?k=Office+Chair&lo=grid&crid=1N60K12GUA798&qid=1601040579&sprefix=chair&ref=sr_pg_1']

    def parse(self, response):
        items = response.css('.s-asin .sg-col-inner')

        for item in items:
            loader = ItemLoader(item=ScrapingamazonItem(), selector=item)
            loader.add_css('ProductName', '.a-color-base.a-text-normal::text')
        
        yield loader.load_item()

I am running it using scrapy crawl amazon_spider -o file.csv. The file returns empty.

Any help is deeply appreciated! :)

Upvotes: 0

Views: 360

Answers (1)

AaronS
AaronS

Reputation: 2335

Try

for item in items:
   loader = ItemLoader(item=ScrapingamazonItem(), selector=item)
   loader.add_css('ProductName', '.a-color-base.a-text-normal::text')
   yield loader.load_item()

Upvotes: 2

Related Questions