CODEWITHSUNDEEP
pythonweb-scrapingscrapy
nam24
nam24

Reputation: 51

Python Scrapy Isn't Extracting Data

Full disclaimer - I'm not a programmer. I'm trying to get the 12 month rent price (which is currently 1,976) by scraping the following webpage - https://www.essexapartmenthomes.com/apartments/bonita-cedars/floor-plans-and-pricing. My problem is that when I enter the below into my shell terminal, no results are being returned even though I expect some sort of information. I thought this would have been relatively straightforward from the tutorials I've watched, but this website looks to be structured differently (perhaps more complex). I used SelectorGadget to verify the CSS Selector is correct. What am I missing?

scrapy shell "https://www.essexapartmenthomes.com/apartments/bonita-cedars/floor-plans-and-pricing"
response.css('.pricing-list::text').extract()

enter image description here

Upvotes: 0

Views: 53

Answers (1)

Krisz
Krisz

Reputation: 2224

It's not going to be that easy since the linked page relies heavily on JavaScript. You have two options:

  • You can use use a rendering engine like splash to render the JavaScript after you load the page and see if you can extract the data
  • Or you can see what endpoints the site uses to fetch the data which you can fetch yourself manually.

Either way, it's not going to be as trivial as you thought and might be a good idea to consult someone with experience.

Upvotes: 1

Related Questions