CODEWITHSUNDEEP
pythoncookiesweb-scrapingscreen-scraping
B.Mr.W.
B.Mr.W.

Reputation: 19648

Send Cookie Using Python Requests

I am trying to scrape some data (reproduce a POST operation I did in a browser) by using Python Requests library. Expecting it will return the content I saw while using browser by copying the request header and post form.

However, I am not quite sure what is the correct way to send cookies using Python Requests. Here is a screen shot how it looks like in Chrome.

It looks like I can either use cookie as a key in the request header or use the cookie argument in the post command.

(1). If I want to use cookie in the request header, should I treat the whole cookie content as a long string and set the key to be cookie and set the value to be that string?

(2). If I want to use the cookie argument in the request.post command, should I manually translate that long string into a dictionary first, and then pass to cookie argument?. Something like this?

 mycookie = {'firsttimevisitor':'N'; 'cmTPSet':'Y'; 'viewType':'List'... }
 # Then 
 r = requests.post(myurl, data=myformdata, cookies=mycookie, headers=myheaders)

Thanks!

Upvotes: 0

Views: 15800

Answers (2)

laike9m
laike9m

Reputation: 19388

Just follow the documentation:

>>> url = 'http://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')

>>> r = requests.get(url, cookies=cookies)
>>> r.text
'{"cookies": {"cookies_are": "working"}}'

So use a dict for cookie, and note it's cookies not cookie.

Upvotes: 3

Vincent Beltman
Vincent Beltman

Reputation: 2114

  1. Yes. But make sure you call it "Cookie" (With capital C)

  2. I always did it with a dict. Requests expects you to give a dict.

A string will give the following

cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
TypeError: string indices must be integers, not str

Upvotes: 2

Related Questions