CODEWITHSUNDEEP
pythonurllib2
Wells
Wells

Reputation: 10969

python try/except with urllib2 throwing odd exception

Function looks like:

def fetchurl(url):
    timeout = 10

    try:
        res = urllib2.urlopen(url, timeout=timeout)
        reader = csv.reader(res)
        reader.next() # Trim the CSV header
        return reader
    except urllib2.URLError, e:
        print 'bailing on %s (timeout of %s exceeded)' % (url, timeout)
        return None

Exception looks like:

  File "scrape.py", line 35, in fetchurl
    reader.next() # Trim the CSV header
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 530, in next
    line = self.readline()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 601, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 647, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
socket.timeout: timed out

Why isn't the try/except block catching the socket.timeout exception?

Upvotes: 0

Views: 1285

Answers (1)

Thunderboltz
Thunderboltz

Reputation: 1627

Because its got nothing to do with urllib2.URLError something to do with the exception within the File "scrape.py".

Error cases within file "scrape.py" which you're using for csv.reader are not handled well.

You use the following as prescribed here:

import socket

try:
    resp = urllib2.urlopen(req, timeout=5)
except urllib2.URLError:
    print "Bad URL or timeout"
except socket.timeout:
    print "socket timeout"

Upvotes: 1

Related Questions