CODEWITHSUNDEEP
scrapyexport-to-csv
whands
whands

Reputation: 21

Scrapy - How to define structure of the csv export (columns,etc..)

I am quite new with scrapy, and i'm to figure out how to set the structure of a csv export. I have in the following example 2 kind of data scraped: ids and links

Here is the simple code i'm using :

class MybotSpider(scrapy.Spider):
    name = 'mybot'
    start_urls = ['url']

    def parse(self, response):
        all_dataids = response.css('li::attr(data-id)').extract()
        all_links = response.xpath('//a[contains(@class, "js_clickable")]/@href').extract()

        adlist = SpiderItem()
        adlist['dataid'] = all_dataids
        adlist['link'] = all_links
        yield adlist

But my export is like this :

csv_export_i_get

instead, i would like to export with for each id corresponding the href and separated in rows :

csv_export_i_would_like

Upvotes: 2

Views: 52

Answers (1)

vezunchik
vezunchik

Reputation: 3717

Make yielding items in this way:

def parse(self, response):
    all_dataids = response.css('li::attr(data-id)').extract()
    all_links = response.xpath('//a[contains(@class, "js_clickable")]/@href').extract()

    for link, dataid in zip(all_links, all_dataids):
        adlist = SpiderItem()
        adlist['dataid'] = dataid
        adlist['link'] = link
        yield adlist

Here you zip your arrays to ((link, dataid), (link, dataid), (link, dataid), ...) and then yielding them one by one. So it should give you desired output.

Upvotes: 1

Related Questions