CODEWITHSUNDEEP
python
wong2
wong2

Reputation: 35760

how to scrape html generated by javascript using python?

I want to scrape the html generated by javascript , just like what you can see in Firebug.

UPDATE: I've found this article: http://blog.motane.lu/2009/07/07/downloading-a-pages-content-with-python-and-webkit/ which use PyQt to solve the problem and it works well for me.

BUT another problem occur: I have to login the website first, but I don't know how to simulate login in PyQt .... :(

Upvotes: 5

Views: 1207

Answers (3)

wisty
wisty

Reputation: 7061

You could use python spidermonkey, which is a python wrapper to Firefox's engine: http://code.google.com/p/python-spidermonkey/

But the project seems a bit immature.

Upvotes: 0

Jaanus
Jaanus

Reputation: 17866

To be precise with terminology, Javascript does not generate HTML. Javascript generates and manipulates the DOM in your browser. The Firebug is showing you HTML representation of that DOM so that it would be readable. The HTML does not actually exist. :)

I don't think an out-of-the box easy solution exists. You may want to look at this blog post and comments that have some pointers.

Upvotes: 0

Oren Hizkiya
Oren Hizkiya

Reputation: 4434

Have a look at this article which describes using Windmill to do scrape a page after Javascript has been executed by the browser.

This article will show how to extract the desired information using the same three steps when the web page is not written directly using HTML, but is auto-generated using JavaScript to update the DOM tree.

They have some examples I am sure you can easily adapt.

Upvotes: 5

Related Questions