CODEWITHSUNDEEP
scrapyweb-crawler
Zarina veeru
Zarina veeru

Reputation: 17

using scrapy creating a spider and unable to store data to csv

I'm pretty new to scrapy, here I created a spider using the amazon URL unable to get the output to the csv.

Here is my code:

import scrapy

class AmazonMotoMobilesSpider(scrapy.Spider):
name = "amazon"
start_urls = ['https://www.amazon.in/Samsung-Mobiles/b/ref=amb_link_47?ie=UTF8&node=4363159031&pf_rd_m=A1VBAL9TL5WCBF&pf_rd_s=merchandised-search-leftnav&pf_rd_r=NGA52N9RAWY1W103MPZX&pf_rd_r=NGA52N9RAWY1W103MPZX&pf_rd_t=101&pf_rd_p=1ce3e975-c6e8-479a-8485-2e490b9f58a9&pf_rd_p=1ce3e975-c6e8-479a-8485-2e490b9f58a9&pf_rd_i=1389401031',]
def parse(self,response):
    product_name = response.xpath('//h2[contains(@class,"a-size-base s-inline  s-access-title  a-text-normal")]/text()').extract()
    product_price = response.xpath('//span[contains(@class,"a-size-base a-color-price s-price a-text-bold")]/text()').extract()
    yield {'product_name'product_name,'product_price': product_price}

My shell is showing this result:

len(response.xpath('//h2[contains(@class,"a-size-base s-inline s-access-tit le a-text-normal")]/text()')) 24

do I need to change any settings ?

Upvotes: 0

Views: 205

Answers (2)

Tarun Lalwani
Tarun Lalwani

Reputation: 146630

To generate results in CSV you need to run the crawler with a output option

scrapy crawl -o results.csv spidername

Only when you activate a output the results are sent to the file. Else they will processed by your piplelines. If you are not saving them anywhere through pipeline then they will be just on terminal's console logs

Upvotes: 1

Arun Augustine
Arun Augustine

Reputation: 1776

I think Its because your yield has some syntax errors in the dictionary.

Change this

yield {'product_name'product_name,'product_price': product_price}

to

yield {'product_name':product_name,'product_price': product_price}

Upvotes: 0

Related Questions