CODEWITHSUNDEEP
pythonweb-scrapingpython-requests
TerenceLam
TerenceLam

Reputation: 173

Response [412] when using the requests python package to access this webpage, how to get around it?

This is the reproducible code:

import requests
url = 'http://wjw.hubei.gov.cn/'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36'}
res = requests.get(url,headers=headers)
print(res)

The code print(res) gives the following output:

<Response [412]>

I can open the webpage fine on my computer with Chrome.

Is there something missing in the header? Is there a way to get around the 412 error? Thanks in advance!

Upvotes: 2

Views: 1865

Answers (2)

user21173219
user21173219

Reputation: 1

import requests response=requests.get("https://precog.iiit.ac.in/") < Response [200] >

<Response [400]>

<Response [800]>

None of the above responses

Upvotes: -1

αԋɱҽԃ αмєяιcαη
αԋɱҽԃ αмєяιcαη

Reputation: 11515

That website require a valid Cookie in order to response back to you.

I've tried several ways such as calling the main website and then retrieving the Cookie under requests.Session() but the website is not allowing me to pass through.

So the only way which you can use as for now. Or to use Selenium or pass a valid Cookie to the requests

Here's how to get the Cookie and User-Agent via the browser:

enter image description here

Using the following Code:

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0",
    "Cookie": "Hm_lvt_5544783ae3e1427d6972d9e77268f25d=1578572654; Hm_lpvt_5544783ae3e1427d6972d9e77268f25d=1578572671; dataHide2=64fa0f2a-a6aa-43b4-adf0-ce901e8d1a37; FSSBBIl1UgzbN7N80S=sXE0qXcyGkTm4uVerLqfZyUU3XFMZzkm22k.eqVABLPe0eYMo3D8uX5ZJ07.7cCr; FSSBBIl1UgzbN7N80T=4aY.P74ZFvDef6i1BgsPAGpjsGOCcIHJFaOyshl4_fJ1WvTk1nqBkdG9PsyX3VRZcIuI8zdYiRJw4rEBQfx.Mv.GS_wT6Hzgiw.AY.UMP.Mw4iCKXGDzY1UeIH2gUd15impxzBVzZpN3MnSdqD0TUqcxSq0RrvIuE8RKT5pFLAqaNnVqtbeSACx43yIYtKJ41y8Isu6a6lNOlWNeaFJ8bx22pKm3lAIO.HIDhGSZqrUP76.q3i4Iux59f7dqJPuSRF90G1LSUBE8t8HrlWzBcSwJJJARX4Ioc0iHmHvdkVoigUitTRjLUHJM4ieOV1sLBDsq"
}

r = requests.get("http://wjw.hubei.gov.cn/", headers=headers)
print(r)

Output:

<Response [200]>

enter image description here

Update:

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"}

with requests.Session() as req:
    r = req.get("http://www.hubei.gov.cn/")
    headers['Cookie'] = r.headers.get("Set-Cookie")
    for item in range(10):
        new = req.get("http://wjw.hubei.gov.cn/", headers=headers)
        print(new)

enter image description here

Upvotes: 4

Related Questions