CODEWITHSUNDEEP
pythonxpathscrapy
Bodhistawa
Bodhistawa

Reputation: 15

Scrapy: how to output in multiple rows for each data

I've written a script in python scrapy, to parse some items from price comparison website. I'm not satisfied with result because script stick all results in one row like:

price,seller
"4,4,4,4,4,4,4,4,4,4","Opinie o ewitaldo.pl,Opinie o dobrazielarnia.pl,Opinie o brokulek.pl,Opinie o delikatesyznatury.pl,Opinie o bialastokrotka.pl,Opinie o aleeko.pl,Opinie o aptekamini.pl,Opinie o farmeko24.pl,Opinie o straganzdrowia.pl,Opinie o bdsklep.pl"

and what I need is:

price,seller
4,Opinie o ewitaldo.pl
4,Opinie o dobrazielarnia.pl
4,Opinie o brokulek.pl
4,Opinie o delikatesyznatury.pl
4,Opinie o bialastokrotka.pl
4,Opinie o aleeko.pl
4,Opinie o aptekamini.pl
4,Opinie o farmeko24.pl
4,Opinie o straganzdrowia.pl
4,Opinie o bdsklep.pl

Here is my spider code:

import scrapy
from ceneo.items import CeneoItem


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'https://www.ceneo.pl/33022301'
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)          

    def parse(self, response):
        items = CeneoItem()
        sale_price = response.xpath('(//td[@class="cell-price"] /a/span/span/span[@class="value"]/text())[position() <= 10]').extract()
        seller = response.xpath('(//tr/td/div/ul/li/a[@class="js_product-offer-link"]/text())[position()<=10]').extract()
        items['product_sale_price'] = ''.join(sale_price).strip()
        items['product_seller'] = ''.join(seller).strip()
        yield {'price': sale_price, 'seller': seller}

What should be changed to get nice csv output?

Upvotes: 1

Views: 1535

Answers (2)

furas
furas

Reputation: 143097

You have to use zip() to group price with seller and then yield every pair separately

def parse(self, response):

    all_prices = response.xpath('(//td[@class="cell-price"] /a/span/span/span[@class="value"]/text())[position() <= 10]').extract()
    all_sellers = response.xpath('(//tr/td/div/ul/li/a[@class="js_product-offer-link"]/text())[position()<=10]').extract()

    for price, seller in zip(all_prices, all_sellers):
        yield {'price': price.strip(), 'seller': seller.strip()}

Upvotes: 4

Arun Augustine
Arun Augustine

Reputation: 1766

You have the sale_price and seller values.

sale_data = [sale_price, seller]
final_list = [zip(*sale_data)[i] for i in xrange(len(sale_price))]

this final_list will disply result likes this.

[(u'2893', u'Opinie o klimasklep.pl'), (u'2999', u'Opinie o wentsystem.com'), (u'3419', u'Opinie o master.sklep.pl'), (u'2893', u'Opinie o elektrozilla.pl'), (u'2895', u'Opinie o mk.net.pl'), (u'2949', u'Opinie o e-master.pl'), (u'2960', u'Opinie o kajt24.pl'), (u'2999', u'Opinie o klimatop.pl'), (u'2999', u'Opinie o fabryka-komfortu.pl'), (u'3041', u'Opinie o pajm.pl')]

Upvotes: -1

Related Questions