CODEWITHSUNDEEP
pythonweb-scraping
Jeff
Jeff

Reputation: 1242

web scraping dynamic content with python

I'd like to use Python to scrape the contents of the "Were you looking for these authors:" box on web pages like this one: http://academic.research.microsoft.com/Search?query=lander

Unfortunately the contents of the box get loaded dynamically by JavaScript. Usually in this situation I can read the Javascript to figure out what's going on, or I can use an browser extension like Firebug to figure out where the dynamic content is coming from. No such luck this time...the Javascript is pretty convoluted and Firebug doesn't give many clues about how to get at the content.

Are there any tricks that will make this task easy?

Upvotes: 6

Views: 21206

Answers (3)

ivan_pozdeev
ivan_pozdeev

Reputation: 35998

For scraping dynamic content, you need not a simple scraper but a full-fledged headless browser.

dhamaniasad/HeadlessBrowsers: A list of (almost) all headless web browsers in existence is the fullest list of these that I've seen; it lists which languages each has bindings for.

(Note that more than a few of the listed projects are abandoned!)

Upvotes: 1

Nick C.
Nick C.

Reputation: 191

Instead of trying to reverse engineer it, you can use ghost.py to directly interact with JavaScript on the page.

If you run the following query in a chrome console, you'll see it returns everything you want.

document.getElementsByClassName('inline-text-org');

Returns

[<div class=​"inline-text-org" title=​"University of Manchester">​University of Manchester​</div>, 
 <div class=​"inline-text-org" title=​"University of California Irvine">​University of California ...​</div>​
  etc...

You can run JavaScript through python in a real life DOM using ghost.py.

This is really cool:

from ghost import Ghost
ghost = Ghost()
page, resources = ghost.open('http://academic.research.microsoft.com/Search?query=lander')
result, resources = ghost.evaluate(
    "document.getElementsByClassName('inline-text-org');")

Upvotes: 10

danstaaar
danstaaar

Reputation: 43

A very similar question was asked earlier here. Quoted is selenium, originally a testing environment for web-apps.

I usually use Chrome's Developer Mode, which IMHO already gives even more details than Firefox.

Upvotes: 3

Related Questions