Codesmith
Codesmith

Reputation: 6752

Python urllib2 - Freezes when connection temporarily dies

So, I'm working with urllib2, and it keeps freezing on a specific page. Not even Ctrl-C will cancel the operation. It's throwing no errors (I'm catching everything), and I can't figure out how to break it. Is there a timeout option for urllib2 that defaults to never?

Here's the procedure:

req = urllib2.Request(url,headers={'User-Agent':'...<chrome's user agent string>...'})
page = urllib2.urlopen(req)
// p.s. I'm not installing any openers

Then, if the internet gets cut partway through the second line (which downloads it), even if connection is restored, this freezes the program completely.

Here's the response header I get in my browser (Chrome) from the same page:

HTTP/1.1 200 OK
Date: Wed, 15 Feb 2017 18:12:12 GMT
Content-Type: application/rss+xml; charset=UTF-8
Content-Length: 247377
Connection: keep-alive
ETag: "00e0dd2d7cab7cffeca0b46775e1be7e"
X-Robots-Tag: noindex, follow
Link: ; rel="https://api.w.org/"
Content-Encoding: gzip
Vary: Accept-Encoding
Cache-Control: max-age=600, private, must-revalidate
Expires: Wed, 15 Feb 2017 18:12:07 GMT
X-Cacheable: NO:Not Cacheable
Accept-Ranges: bytes
X-Served-From-Cache: Yes
Server: cloudflare-nginx
CF-RAY: 331ab9e1443656d5-IAD

p.s. The url is to a large WordPress feed which, according to the response, appears compressed.

Upvotes: 0

Views: 87

Answers (1)

sxn
sxn

Reputation: 508

According to the docs, the default timeout is, indeed, no timeout. You can specify a timeout when calling urlopen though. :)

page = urllib2.urlopen(req, timeout=30)

Upvotes: 2

Related Questions