CODEWITHSUNDEEP
pythonfirebugweb-scraping
Tom
Tom

Reputation: 91

Scraping a website in python with firebug?

I am trying to scrape a website, but the thing that I want to get is not in the source code. But it does appear when i use firebug. Is there a way to scrape from the firebug code as opposed to the source code?

Upvotes: 0

Views: 1981

Answers (2)

P Daddy
P Daddy

Reputation: 29537

What you're after is probably loaded dynamically by JavaScript (possibly with Ajax). To grab it programmatically outside of the browser, you'd have to render the JavaScript, which is not easy. You should be able to run a Greasemonkey script to scrape it after rendering, though. See http://blog.nparashuram.com/2009/08/screen-scraping-with-javascript-firebug.html.

Upvotes: 1

Managu
Managu

Reputation: 9039

If the answer's not in the source code (possibly obfuscated, encoded, etc), then it was probably retrieved after the page loaded with an XmlHTTPRequest. You can use the 'network' panel in Firebug to see what other pieces of data the page loaded, and what requests it made to load them.

(You may have to enable the network panel and then reload the page/start over)

Upvotes: 0

Related Questions