CODEWITHSUNDEEP
pythonsessionpython-requests
jim jarnac
jim jarnac

Reputation: 5152

how to use requests.session so that headers are presevred and reused in subsequent get requests

I might have misunderstood the requests.session object.

headers ={'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}
s = requests.Session()
r = s.get('https://www.barchart.com/', headers = headers)
print(r.status_code)

This works fine and return 200 as expected.

However this following return 403 and shows that the headers from the first request have not been saved in the session like it would manually using a browser:

headers ={'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}
s = requests.Session()
r = s.get('https://www.barchart.com/', headers = headers)
r = s.get('https://www.barchart.com/futures/quotes/CLQ20')
print(r.status_code)
print(s.headers)

I thought there would be a way to compound headers, cookies etc from 1 requests to the other using the session object... am i wrong ?

Upvotes: 15

Views: 25478

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195438

You can use session.headers (doc) property to specify headers that are sent with each request:

import requests

headers ={'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}

s = requests.session()
s.headers = headers       # <-- set default headers here

r = s.get('https://www.barchart.com/')
print(r.status_code)
print(s.headers)
print('-' * 80)
r = s.get('https://www.barchart.com/futures/quotes/CLQ20')
print(r.status_code)
print(s.headers)

s.close()

Prints:

200
{'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}
--------------------------------------------------------------------------------
200
{'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}

Upvotes: 20

Related Questions