CODEWITHSUNDEEP
pythonurlliburlopen
narnie2
narnie2

Reputation: 1

Python: urlopen not downloading the entire site

Greetings,

I have done:

import urllib

site = urllib.urlopen('http://www.weather.com/weather/today/Temple+TX+76504')
site_data = site.read()
site.close()

but it doesn't compare to viewing the source when loaded in firefox.

I suspected the user agent and did this:

class AppURLopener(urllib.FancyURLopener):
    version = "Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.2.8) Gecko/20100722 Ubuntu/10.04 (lucid) Firefox/3.6.8"

urllib._urlopener = AppURLopener()

and downloaded it, but it still doesn't download the whole website.

Can someone please help me do user agent switching, if that is the likely culprit?

Thanks, Narnie

Upvotes: 0

Views: 1385

Answers (2)

mykhal
mykhal

Reputation: 19905

downloaded page displayed locally may look different from several reasons, like that there are relative links (can be fixed adding e.g. <base href="http://www.weather.com/today/"> into the page head element), or non-functional ajax requests (see Ways to circumvent the same-origin policy).

Upvotes: 2

aaronasterling
aaronasterling

Reputation: 70984

It's more likely that there is an iframe in the code or that javascript is modifying the DOM. If theres an iframe, you'll have to parse the page to get the url for the iframe or just do it manually if it's a one-off. If it's javascript, I hear that selenium-rc is good but have no first hand experience with it.

Upvotes: 3

Related Questions