CODEWITHSUNDEEP
pythonweb-scrapingurllib
Alexis Eggermont
Alexis Eggermont

Reputation: 8145

urllib slower than browser to access html

The following python script takes 3 seconds on my PC to load the source code of a twitter page, which is much higher than it takes to retrieve the source code of other websites, such as youtube. When I load the same twitter page in my browser, the "network" tab in google chrome tells me the html is retrieved in 0.3 seconds.

Why is urllib so much slower than my browser?

import urllib2
import time

start=time.time()
channel='pontifex'
url="https://twitter.com/"+channel
page = urllib2.urlopen(url).read()
print str(round(time.time()-start,0))+" secs total"

Upvotes: 3

Views: 1066

Answers (1)

smac89
smac89

Reputation: 43088

Caching is the answer and it is usually done by browsers to reduce the load time of frequently visited sites. If not the browser, then search engines such as Google also cache frequently visited websites so that retrieving them is only a matter of milliseconds

See this post: How can Google be so fast?

Upvotes: 2

Related Questions