CODEWITHSUNDEEP
javascriptpythonhtmlweb-scrapingpython-requests
gstvolvr
gstvolvr

Reputation: 650

How to access HTML of webpage after button press

I am trying to scrape some data from the following web page:

College Board - Georgia Institute of Technology

But the information I need to access is only displayed after pressing the "Applying" tab on the left. Since the URL does not change, how can I simulate pressing the button in order to scrape the HTML?

I am using Python3.3 and the requests module.

Upvotes: 1

Views: 169

Answers (1)

alecxe
alecxe

Reputation: 474141

According to the page source, the information you need is hidden inside a javascript code and is calculated and rendered after the click on "Applying" link.

requests simply cannot make in-browser user actions and, since there is no additional requests going after clicking "Applying", you cannot get the data without actually having a real browser to run that js code. Mechanize also wouldn't help because it cannot handle js.

Consider using selenium (FYI, you can also use a headless PhantomJS browser).

Hope that helps.

Upvotes: 1

Related Questions