CODEWITHSUNDEEP
pythonxpathscrapy
PDog
PDog

Reputation: 143

Scrapy xpath selector not parsing

I'm trying to scrape https://www.grailed.com/ using scrapy. I have been able to get the elements I want in each listing (price, item title, size). I currently trying to get the ahrefs for each listing at the home page.

When I try response.xpath('.//div[starts-with(@id, "product")]').extract returns

<bound method SelectorList.extract of [<Selector xpath='.//div[starts-with(@id, "product")]' 
data=u'<div id="products">\n<div id="loading">\n<'>]>

Based on the inspect element it should be returning div class="feed-wrapper">?

I'm just trying to get those links so scrapy knows to go into each listing. Thank you for any help.

Upvotes: 0

Views: 738

Answers (1)

Dmytro Pastovenskyi
Dmytro Pastovenskyi

Reputation: 5429

When you do scrapping always check source of page (not in the inspector but view-source) - that would be real data you operate with.

That div is added dynamically after page loads. JS does that job. When you send request to server and receive pure HTML - JS will not be executed and so you see real Server response which you support to work with.

div class="feed-wrapper">

Here is real Server response to you. You must deal with it.

enter image description here

Upvotes: 1

Related Questions