CODEWITHSUNDEEP
pythonurllib2urllib
Florian Mayer
Florian Mayer

Reputation: 3081

Get socket for urllib2.urlopen return value for HTTP

I'm trying to do asynchronous downloading of files using urllib2 but have no succeeded in finding out the socket (or its fileno) to wait for new data for for HTTP requests. Here's what I've already tried.

>>> from urllib2 import urlopen
>>> from select import select
>>> r = urlopen('http://stackoverflow.com/')
>>> select([r], [], [])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/socket.py", line 307, in fileno
    return self._sock.fileno()
AttributeError: HTTPResponse instance has no attribute 'fileno'
>>> r.fileno()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/socket.py", line 307, in fileno
    return self._sock.fileno()
AttributeError: HTTPResponse instance has no attribute 'fileno'
>>> r.fp.fileno()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/socket.py", line 307, in fileno
    return self._sock.fileno()
AttributeError: HTTPResponse instance has no attribute 'fileno'
>>> select([r.fp], [], [])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/socket.py", line 307, in fileno
    return self._sock.fileno()
AttributeError: HTTPResponse instance has no attribute 'fileno'
>>>

Upvotes: 5

Views: 2548

Answers (1)

agf
agf

Reputation: 176930

See http://www.velocityreviews.com/forums/t512553-re-urllib2-urlopen-broken.html.

The problem is that urlib2 was changed to wrap an HTTPResponse object in a socket._fileobject to get a few more file methods. Except (as reported above) HTTPResponse doesn't have a fileno() method, so when _fileobject tries to use it, it blows up.

The solution

Adding an appropriate method to HTTPResponse:

def fileno(self):
    return self.fp.fileno()

Or, alternatively, use urllib.urlopen instead of urrlib2.urlopen.

There is a bug report for this issue; it was fixed in Python 3 and in Python 2.7.

Upvotes: 3

Related Questions