CODEWITHSUNDEEP
pythonhtmlajaxpython-2.7urllib2
Kunc
Kunc

Reputation: 102

Python urllib2 - get html after an Ajax call

I am designing as part of my python program a scraper that needs to be able to obtain the result of an Ajax call. For example, when I go on Google chrome, after inspecting an element I can see the html for the website, but when viewing the source in general there are only scripts to create that code. To do this I would much prefer to use urllib2 as this is the library I have used throughout the program. I use python 2.7 and intend to distribute it using py2exe if that makes a difference.

Upvotes: 0

Views: 676

Answers (1)

mcepl
mcepl

Reputation: 2786

This cannot be done, unless you somehow manage to run the webpage’s scripts through a true JavaScript interpreter and download all required resources.

Note, however, that some websites provide downgraded version of their content, either for non-GUI browsers (e.g., lynx widely used by visually impaired users) or for search robots. See for example https://developers.google.com/webmasters/ajax-crawling/docs/getting-started for one way how it is done (on almost many Google websites, among many other ones).

Upvotes: 1

Related Questions