CODEWITHSUNDEEP
javascriptpythonhtmlweb-scraping
toolshed
toolshed

Reputation: 2019

Why is requests.get() retrieving different HTML using Python than browser?

I am attempting to extract data from an HTML table, but it appears that the HTML isn't loading correctly when using requests.get(). Instead, a line in the source reads:

"JavaScript is not enabled and therefore this page may not function correctly."

When I navigate to the page in Google Chrome, the HTML appears as it should.

How do I get a Python script to load the proper HTML?

Upvotes: 3

Views: 5492

Answers (2)

alecxe
alecxe

Reputation: 473833

Welcome to the wonderful world of web-crawling. The problem you are experiencing is that requests.get() would just get you the initial page that the browser receives at the beginning of a page load. But, this is not the page you see in the browser since there could be so much involved in forming the web page: javascript function calls, AJAX calls etc.

If you want to programmatically get the HTML you see when you click "Show source" in a web browser after the page was loaded - you would need a real browser. This is there selenium could be a good option:

from selenium import webdriver

browser = webdriver.Firefox()
browser.get(url)
print browser.page_source

Note that selenium itself is very powerful in terms of locating elements - you don't need a separate HTML parser for extracting the data out of the page.

Hope that helps.

Upvotes: 9

Saren Arterius
Saren Arterius

Reputation: 1340

If you are sure that you have to deal with JavaScript, webdriver will handle better and saves your life.

from selenium.common.exceptions import NoSuchElementException
from selenium import webdriver
from time import sleep

browser = webdriver.Firefox()
browser.get("http://yourwebsite.com/html-table")
browser.find_element_by_id("some-js-triggering-elem").click()
while 1:
    try:
        browser.find_element_by_id("elem-that-makes-you-know-that-table-is-loaded")
    except NoSuchElementException:
        sleep(1)
html = browser.find_element_by_xpath("//*").get_attribute("outerHTML")
# Use PyQuery or something else to parse the html and get data from table

Upvotes: 0

Related Questions