Leonardo Quintero
Leonardo Quintero

Reputation: 7

I think I have the xpaths right but keep getting "Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)"

I'm trying to scrape Yelp with Scrapy and everything seems to be just fine. I even made sure that the url and all xpaths work in Scrapy shell properly but I keep getting: Crawled 0 pages (at 0 pages/min), and does not give an error I can fix.

I also tried looking for similar problems but nothing seems to be related to what's happening to my spider.

from scrapy import Spider
from ..items import YelpItem
import scrapy
import re 

class YelpSpider(Spider):
    name = "yelp"
    allowed_domains = ['www.yelp.com']
    # Defining the list of pages to scrape
    start_url = ['https://www.yelp.com/search?find_desc=Dog&find_loc=Boston%2C%20MA&start=0' + str(1 * i)  for i in range(0, 3)]

def parse(self, response):
    # Defining rows to be scraped
    rows = response.xpath('//[@id="wrap"]/div[3]/div[2]/div[2]/div/div[1]/div[1]/div/ul/li').extract_first()
    for row in rows:
        # Scraping Busines' Name
        name = row.xpath('.//p/a/text()').extract_first()

        # Scraping Phone number
        phone = rows.xpath('.//div[1]/p[1][@class= "lemon--p__373c0__3Qnnj text__373c0__2pB8f textcolor--normal__373c0__K_MKN text-align--right__373c0__3ARv7"]/text()').extract_first()

        # scraping area
        area = rows.xpath('.//p/span[@class = "lemon--span__373c0__3997G"]/text()').extract_first()


        item = YelpItem()    
        item['name'] = name
        item['phone'] = phone
        item['area'] = area

        yield item

Upvotes: 0

Views: 49

Answers (1)

gangabass
gangabass

Reputation: 10666

You need to change your start_url to start_urls.

Upvotes: 1

Related Questions