CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Dave
Dave

Reputation: 95

Not able to load and scrape data from webpage using Scrapy

I'm trying to scrape data from https://www.grailed.com/ using the Scrapy framework in Python, but when I'm in the 

scrapy shell

and trying to learn how to extract the data, with

response.css("my css path")

or

response.xpath("my xpath")

I always get empty lists. Then, when I fetch and view the response in the shell for https://www.grailed.com, I get an almost blank page that does not contain the data that I would like to scrape (items, price, etc). In this case, would Scrapy still work for my purposes? If not, are there any alternatives? Thanks!

Upvotes: 0

Views: 1753

Answers (1)

JBJ
JBJ

Reputation: 1109

This site loads content using JS. Scrapy does not support JS. That is why you see empty html template without data itself. Possible solutions:

  1. Try using headless browser that will execute all JS on the page. At times I find phantomjscloud very handy API solution for rendering JS sites. You can use like this - enter link description here
  2. Check network conenctions tab to see if there are some API calls that return data you need in json format, so you can get it directly.Like this one

Upvotes: 2

Related Questions