CODEWITHSUNDEEP
pythontwisted
grromrell
grromrell

Reputation: 286

Making Head Requests in Twisted

I am relatively new to using Twisted and I am having trouble returning the content-length header when performing a basic head request. I have set up an asynchronous client already but the trouble comes in this bit of code:

def getHeaders(url):
    d = Agent(reactor).request("HEAD", url)
    d.addCallbacks(handleResponse, handleError)
    return d

def handleResponse(r):
    print r.code, r.headers

    whenFinished = twisted.internet.defer.Deffered()
    r.deliverBody(PrinterClient(whenFinished))

    return whenFinished

I am making a head request and passing the url. As indicated in this documentation the content-length header is not stored in self.length, but can be accessed from the self.headers response. The output is returning the status code as expected but the header output is not what is expected. Using "uhttp://www.espn.go.com" as an example it currently returns:

Set-Cookie: SWID=77638195-7A94-4DD0-92A5-348603068D58; 
path=/; expires=Fri, 31-Jan-2034 00:50:09 GMT; domain=go.com;
X-Ua-Compatible: IE=edge,chrome=1
Cache-Control: max-age=15
Date: Fri, 31 Jan 2014 00:50:09 GMT
P3P: CP="CAO DSP COR CURa ADMa DEVa TAIa PSAa PSDa IVAi IVDi CONi 
OUR SAMo OTRo BUS PHY  ONL UNI PUR COM NAV INT DEM CNT STA PRE"
Content-Type: text/html; charset=iso-8859-1

As you can see, no content-length field is returned. If the same request is done in requests then the result will contain the content-length header:

r = requests.head("http://www.espn.go.com")
r.headers
({'content-length': '46133', 'content-encoding': 'gzip'...})
(rest omitted for readability)

What is causing this problem? I am sure it is a simple mistake on my part but I for the life of me cannot figure out what I have done wrong. Any help is appreciated.

Upvotes: 1

Views: 393

Answers (2)

Jean-Paul Calderone
Jean-Paul Calderone

Reputation: 48335

http://www.espn.go.com/ returns one response if the client sends an Accept-Encoding: gzip header and another response if it doesn't.

One of the differences between the two responses is the inclusion of the Content-Length header.

If you want to make requests using Agent including Accept-Encoding: gzip then take a look at ContentDecoderAgent or the third-party treq package.

Upvotes: 3

SingleNegationElimination
SingleNegationElimination

Reputation: 156158

http allows (but does not REQUIRE) entity headers in responses to HEAD requests. The only restriction it places is that 200 responses to HEAD requests MUST NOT include an entity payload. Its up to the origin server to decide which, if any entity headers it would like to include.

In the case of Content-Length, it makes sense for this to be optional for HEAD; if the entity will be computed dynamically (as with compressing/decompressing content), it's better for the server to avoid the extra work of computing the content length when the request won't include the content anyway.

Upvotes: 2

Related Questions