CODEWITHSUNDEEP
pythonpython-requests
Stefan
Stefan

Reputation: 9339

Python requests can't send multiple headers with same key

I'm trying to send a GET request to a server with two headers which have the same name, but different values:

url = 'whatever'
headers = {'X-Attribute': 'A', 'X-Attribute': 'B'}
requests.get(url, headers = headers)

This obviously doesn't work, since the headers dictionary can't contain two keys X-Attribute.

Is there anything I can do, i.e. can I pass headers as something other than a dictionary? The requirement to send requests in this manner is a feature of the server, and I can't change it.

Upvotes: 13

Views: 12587

Answers (4)

Caio Brandão
Caio Brandão

Reputation: 31

url = 'whatever'
headers = {'X-Attribute': "A,B"}
requests.get(url, headers = headers)

Upvotes: 2

Ian Stapleton Cordasco
Ian Stapleton Cordasco

Reputation: 28835

Requests now stores all headers (sent and received) as case insensitively in dictionaries. Beyond even that, though, open up a python console and write:

headers = {'X-Attribute':'A', 'X-Attribute':'B'}

What you get is undefined behaviour. (It may seem reproducible but it is wholly undefined.) So what you're really sending to requests in that instance is this:

{'X-Attribute': 'A'}  # or {'X-Attribute': 'B'}, we can never be certain which it will be

What you could try (but won't work) is:

headers = [('X-Attribute', 'A'), ('X-Attribute', 'B')]

But at least this will be wholly defined behaviour (you'll always send B). As @mata suggested, if your server is HTTP/1.1 compliant, what you can do is this:

import collections

def flatten_headers(headers):
    for (k, v) in list(headers.items()):
        if isinstance(v, collections.Iterable):
           headers[k] = ','.join(v)

headers = {'X-Attribute': ['A', 'B', ..., 'N']}
flatten_headers(headers)
requests.get(url, headers=headers)

Upvotes: 3

mata
mata

Reputation: 69082

requests stores the request headers in a dict, which means every header can only appear once. So without making changes to the requests library itself it won't be possible to send multiple headers with the same name.

However, if the server is HTTP1.1 compliant, it must be able to accept the same as one header with a comma separated list of the single values.

Late edit:
Since this was brought back to my attention, a way to make this work would be to use a custom instance of str that allows to store multiple of the same value in a dict by implementing the hash contract differently (or actually in a CaseInsensitiveDict, wich uses lower() on the names). Example:

class uniquestr(str):

    _lower = None

    def __hash__(self):
        return id(self)

    def __eq__(self, other):
        return self is other

    def lower(self):
        if self._lower is None:
            lower = str.lower(self)
            if str.__eq__(lower, self): 
                self._lower = self
            else:
                self._lower = uniquestr(lower)
        return self._lower

r = requests.get("https://httpbin.org/get", headers={uniquestr('X'): 'A',
                                                     uniquestr('X'): 'B'})
print(r.text)

Produces something like:

{
  ...
  "headers": {
    ...
    "X": "A,B",
  }, 
  ...
}

Interestingly, in the response the headers are combined, but they are really sent as two separate header lines.

Upvotes: 12

user1171968
user1171968

Reputation: 177

requests is using urllib2.urlencode under the hood (or something similar) in order to encode the headers.

This means that a list of tuples can be sent as the payload argument instead of a dictionary, freeing the headers list from the unique key constraint imposed by the dictionary. Sending a list of tuples is described in the urlib2.urlencode documentation. http://docs.python.org/2/library/urllib.html#urllib.urlencode

The following code will solve the problem without flattening or dirty hacks:

url = 'whatever'
headers = [('X-Attribute', 'A'),
           ('X-Attribute', 'B')]
requests.get(url, headers = headers)

Upvotes: 5

Related Questions