CODEWITHSUNDEEP
python-3.xpython-requests-html
rohan_aggarwal
rohan_aggarwal

Reputation: 93

Requests-html package does not render properly for fast.com

I am developing a web-scraping app using python 3.7. I am using requests-html for parsing data. Up until now, I have tried the following code which tries to use the render function (since speed data on fast.com is loaded through javascript).

from requests_html import HTMLSession
quote_page = 'https://fast.com'
session = HTMLSession()
r = session.get(quote_page)
r.html.render()
extract_value = r.html.find('#speed-value', first=True)
print(extract_value.text)

speed-value is the id attribute used by the div that contains the speed data.

But it still prints speed-value as 0.

Upvotes: 3

Views: 2005

Answers (1)

bruce
bruce

Reputation: 482

Speed test need a few seconds, you need wait a few seconds after initial render, so specify a sleep param to render function, which means how many long to sleep after initial render.

For example: Modify r.html.render() to r.html.render(sleep=10)

Tested in my PC,I got a speed number.

Upvotes: 9

Related Questions