CODEWITHSUNDEEP
javascriptpythonhttpcaptcha
Ash
Ash

Reputation: 3550

Python Requests response is different from chrome response

I need to download about 100 captcha images from a particular website. My code in summary is:

1- download the page

2- search for the captcha image URL (using re) and download it

3- :( the downloaded image is different from what I'd see in browser. I guess there is a parameter in session or in the request (get or post) I need to set, which I haven't.

import requests
import re
import time
s = requests.Session()
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

#download this page and look for the url of the captcha image
page = s.get('http://www.rrk.ir/News/ShowOldNews.aspx?Code=1', headers=headers)
result = re.search('img id="imgCaptcha" src="..(.*)"', page.content.decode('utf-8'))
img_url = 'http://www.rrk.ir' + result.group(1).split('"')[0]

print(img_url)
#download the image and save it to a file
img = s.get(img_url, headers=headers)
img_file_name =  './a'  + '.jpg'
with open(img_file_name, 'wb') as fout:
    fout.write(img.content)

s.close()
#:( the downloaded file is different from what I see in Chrome.

How can I find out what setting I'm missing?

Update 1: As suggested, added the custom headers but it didn't help.

Upvotes: 4

Views: 1810

Answers (1)

Denis
Denis

Reputation: 73

Now I'm struggling with a similar problem with authorization. If you have similar problems, you need to turn off automatic redirect in the Requests(allow_redirects=False) and check if there is a cascade of requests in the browser developer tools - perhaps the first request causes redirects between which additional parameters are generated in data/json new payload, or create new headers/cookies..

s = requests.Session()
resp = s.get(url_here, headers=headers, allow_redirects=False)
if resp.status_code == 302:
    print('Redirect!', resp.headers. resp.cookies, sep="\n")

Сheck cookies and response headers! Also you can use:

print(resp.history)

Upvotes: 2

Related Questions