Reputation: 3
from scrapy.spiders import Spider from ..items import QtItem
class QuoteSpider(Spider):
name = 'acres'
start_urls = ['any_url']
def parse(self, response):
items = QtItem()
all_div_names = response.xpath('//article')
for bks in all_div_names:
name = all_div_names.xpath('//span[@class="css-fwbz9r"]/text()').extract()
price = all_div_names.xpath('//h2[@class="css-yr18fa"]/text()').extract()
sqft = all_div_names.xpath('//div[@class="css-1ty8tu4"]/text()').extract()
bhk = all_div_names.xpath('//a[@class="css-163eyf0"]/text()').extract()
yield {
'ttname': name,
'ttprice': price,
'ttsqft': sqft,
'ttbhk': bhk
}
the question has been answered
Upvotes: 0
Views: 67
Reputation: 2335
.//
instead of //
for each variable you're looping overbks
instead of all_div_names
.get()
instead of extract()
as it's one item within the span. get()
grabs a single item, extract()
grabs multiple items.eg. name = bks.xpath('.//span[@class="css-fwbz9r"]/text()').get()
.//
traverses all child elements of all_div_names
XPATH selector. Should always use .//
when you're looping over an XPATH selector with multiple items such as all_div_names
.
eg name = bks.xpath('.//span[@class="css-fwbz9r"]/text()').get()
You will access all span elements of bks in this XPATH selector by using .//
.getall()
instead of extract()
and get()
instead of extract_first()
. With get()
you will always get a string, with extract()
you wont know if you're getting a list or string unfortunately!def parse(self, response):
items = QtItem()
all_div_names = response.xpath('//article')
for bks in all_div_names:
name = bks.xpath('.//span[@class="css-fwbz9r"]/text()').get()
price = bks.xpath('.//h2[@class="css-yr18fa"]/text()').get()
sqft = bks.xpath('.//div[@class="css-1ty8tu4"]/text()').get()
bhk = bks.xpath('.//a[@class="css-163eyf0"]/text()').get()
yield {
'ttname': name,
'ttprice': price,
'ttsqft': sqft,
'ttbhk': bhk
}
Upvotes: 0
Reputation: 2619
You use a for loop but not use for loop variable 'bks'.
for bks in all_div_names:
name = bks.xpath('//span[@class="css-fwbz9r"]/text()').extract()
price = bks.xpath('//h2[@class="css-yr18fa"]/text()').extract()
sqft = bks.xpath('//div[@class="css-1ty8tu4"]/text()').extract()
bhk = bks.xpath('//a[@class="css-163eyf0"]/text()').extract()
Here is our output.
{'ttname': ['Jodhpur Village, Jodhpur, Ahmedabad', 'Shapers Swastik Platinum, Narolgam, Ahmedabad', 'Gayatri Maitri Lake View, Zundal, Ahmedabad', 'Puspak Platinum , Ambli, Ahmedabad', 'arjun greens, Naranpura, Ahmedabad', 'Aariyana Lakeside, Shilaj, Ahmedabad', 'Ganesh Malabar County II, Chharodi, Ahmedabad', 'Jodhpur Village, Jodhpur, Ahmedabad', 'Ratna Paradise, Khoraj, Ahmedabad', 'Thaltej, Ahmedabad', 'Binori Solitaire, Bopal, Ahmedabad', 'Arvind & Safal Parishkaar Apartments, Amraiwadi, Ahmedabad', 'Siddhivinayak Omkar Lotus, Chandkheda, Ahmedabad', 'Orchid Whitefield , Prahlad Nagar, Ahmedabad', 'VISHWAS CITY , Gota, Ahmedabad', 'Gala Aria, Bopal, Ahmedabad', 'Ganesh Malabar County, Chharodi, Ahmedabad', 'Devnandan Infinity , Motera, Ahmedabad', 'Sapphire Swapneel Elysium, Bopal, Ahmedabad', 'Veer Mahavir Hills 2, Koba, Ahmedabad'], 'ttprice': ['₹95.0 L', '₹17.0 L', '₹28.75 L', '₹1.4 Cr', '₹1.0 Cr', '₹3.5 Cr', '₹43.0 L', '₹47.5 L', '₹1.55 Cr', '₹65.0 L', '₹1.1 Cr', '₹42.0 L', '₹74.0 L', '₹50.0 L', '₹30.0 L', '₹1.18 Cr', '₹47.0 L', '₹50.0 L', '₹81.0 L', '₹33.0 L'], 'ttsqft': ['1750 sq.ft', '₹5.43 K/sq.ft', '870 sq.ft', '₹1.95 K/sq.ft', '1125 sq.ft', '₹2.56 K/sq.ft', '2250 sq.ft', '₹6.22 K/sq.ft', '1812 sq.ft', '₹5.52 K/sq.ft', '4275 sq.ft', '₹8.19 K/sq.ft', '1170 sq.ft', '₹3.67 K/sq.ft', '1200 sq.ft', '₹3.96 K/sq.ft', '3340 sq.ft', '₹4.64 K/sq.ft', '1710 sq.ft', '₹3.80 K/sq.ft', '2214 sq.ft', '₹4.97 K/sq.ft', '1108 sq.ft', '₹3.79 K/sq.ft', '1960 sq.ft', '₹3.77 K/sq.ft', '1050 sq.ft', '₹4.76 K/sq.ft', '954 sq.ft', '₹3.14 K/sq.ft', '2115 sq.ft', '₹5.58 K/sq.ft', '1168 sq.ft', '₹4.02 K/sq.ft', '1323 sq.ft', '₹3.78 K/sq.ft', '1800 sq.ft', '₹4.50 K/sq.ft', '1215 sq.ft', '₹2.72 K/sq.ft'], 'ttbhk': ['3 BHK Apartment', '2 BHK Apartment', '2 BHK Apartment', '3 BHK Apartment', '3 BHK Apartment', '4 BHK Apartment', '2 BHK Apartment', '2 BHK Apartment', '4 BHK Apartment', '3 BHK Apartment', '3 BHK Apartment', '2 BHK Apartment', '3 BHK Apartment', '2 BHK Apartment', '2 BHK Apartment', '3 BHK Apartment', '2 BHK Apartment', '2 BHK Apartment', '3 BHK Apartment', '2 BHK Apartment'],
Upvotes: 1