Reputation: 7
I'm trying to scrape Yelp with Scrapy and everything seems to be just fine. I even made sure that the url and all xpaths work in Scrapy shell properly but I keep getting: Crawled 0 pages (at 0 pages/min), and does not give an error I can fix.
I also tried looking for similar problems but nothing seems to be related to what's happening to my spider.
from scrapy import Spider
from ..items import YelpItem
import scrapy
import re
class YelpSpider(Spider):
name = "yelp"
allowed_domains = ['www.yelp.com']
# Defining the list of pages to scrape
start_url = ['https://www.yelp.com/search?find_desc=Dog&find_loc=Boston%2C%20MA&start=0' + str(1 * i) for i in range(0, 3)]
def parse(self, response):
# Defining rows to be scraped
rows = response.xpath('//[@id="wrap"]/div[3]/div[2]/div[2]/div/div[1]/div[1]/div/ul/li').extract_first()
for row in rows:
# Scraping Busines' Name
name = row.xpath('.//p/a/text()').extract_first()
# Scraping Phone number
phone = rows.xpath('.//div[1]/p[1][@class= "lemon--p__373c0__3Qnnj text__373c0__2pB8f textcolor--normal__373c0__K_MKN text-align--right__373c0__3ARv7"]/text()').extract_first()
# scraping area
area = rows.xpath('.//p/span[@class = "lemon--span__373c0__3997G"]/text()').extract_first()
item = YelpItem()
item['name'] = name
item['phone'] = phone
item['area'] = area
yield item
Upvotes: 0
Views: 49