CODEWITHSUNDEEP
pythoncsvscrapy
SIM
SIM

Reputation: 22440

Scrapy is done running with results in the console but CSV output remains blank

I'm very new to scrapy so it's hard for me to find out what i am doing wrong in case of having no results in csv file. I can see results in the console though. Here is what I tried with:

Main folder is named "realyp". Spider file is named "yp.py" and the code: 

from scrapy.selector import Selector
from scrapy.spider import BaseSpider
from realyp.items import RealypItem

class MySpider(BaseSpider):
     name="YellowPage"
     allowed_domains=["yellowpages.com"]
     start_urls=["https://www.yellowpages.com/search?search_terms=Coffee%20Shops&geo_location_terms=Los%20Angeles%2C%20CA&page=2"]

     def parse(self, response):
        title = Selector(response)
        page=title.xpath('//div[@class="info"]')
        items = []
        for titles in page:
            item = RealypItem()
            item["name"] = titles.xpath('.//span[@itemprop="name"]/text()').extract()
            item["address"] = titles.xpath('.//span[@itemprop="streetAddress" and @class="street-address"]/text()').extract()
            item["phone"] = titles.xpath('.//div[@itemprop="telephone" and @class="phones phone primary"]/text()').extract()
            items.append(item)
        return items

"items.py" file includes:

from scrapy.item import Item, Field
class RealypItem(Item):
    name= Field()
    address = Field()
    phone= Field()

To get the csv output my command line is:

cd desktop
cd realyp
scrapy crawl YellowPage -o items.csv -t csv

Any help will be greatly appreciated.

Upvotes: 1

Views: 684

Answers (1)

Tomark
Tomark

Reputation: 369

As stated by @Granitosauros, you should use yield instead of return. The yield should be inside the for cycle. In the for cycle, if the path starts with // then all elements in the document which fulfill following criteria are selected (see here).

Here's a (rough) code that works for me:

# -*- coding: utf-8 -*-
from scrapy.selector import Selector
from scrapy.spider import BaseSpider
from realyp.items import RealypItem

class MySpider(BaseSpider):
    name="YellowPage"
    allowed_domains=["yellowpages.com"]
    start_urls=["https://www.yellowpages.com/search?search_terms=Coffee%20Shops&geo_location_terms=Los%20Angeles%2C%20CA&page=2"]

    def parse(self, response):
        for titles in response.xpath('//div[@class = "result"]/div'):
            item = RealypItem()
            item["name"] = titles.xpath('div[2]/div[2]/h2 /a/span[@itemprop="name"]/text()').extract()
            item["address"] = titles.xpath('string(div[2]/div[2]/div/p[@itemprop="address"])').extract()
            item["phone"] = titles.xpath('div[2]/div[2]/div/div[@itemprop="telephone" and @class="phones phone primary"]/text()').extract()
            yield item

Upvotes: 2

Related Questions