CODEWITHSUNDEEP
pythonhtmlscreen-scrapinghidden
seb
seb

Reputation: 105

Hidden parts in html source code while scraping (python)

So I want to scrape the 'Buy price' integer from this url: https://rsbuddy.com/exchange?id=5502

But when I look at the source code, I can't reach those prices. Neither does BeautifulSoup scraper. This is the ouput from BeautifulSoup:

<div class="col-md-7" id="buy-price">
    ---
</div>

But when I 'inspect element' using chrome, i actually am able to see that price:

<div id="buy-price" class="col-md-7">29,990 gp</div>

Why is that part of the code 'hidden'? Is it simply because they don't want people to scrape from their website? Is there a way to get around this?

Thanks in advance

EDIT: I found the answer by tracking the javascript traffic using the chrome tools. Apparently even though api.rsbuddy.com doesn't give you anything, it does use the api: https://api.rsbuddy.com/grandExchange?a=guidePrice&i=5502

Upvotes: 7

Views: 3437

Answers (2)

duFF
duFF

Reputation: 622

If certain parts of the page are inserted via JavaScript your best bet is to use something like selenium with PhantomJS as the driver.

The Python binding's are quite easy to use and this will allow the JavaScript to execute in the browser and you can grab the prices from there.

Let me know if you want any more information and I'd be happy to help.

Upvotes: 3

CrazyCasta
CrazyCasta

Reputation: 28362

The prices are presumably being put in there by JavaScript. Likely they're using some sort of AJAX to get the prices. You'll have to investigate their JavaScript to get the data you want.

Just to clarify, it's not "hidden" per se, it's just not in the HTML. When you do inspect element it looks at the document consisting of the HTML where it started and any changes the JavaScript has made to it.

Upvotes: 1

Related Questions