CODEWITHSUNDEEP
python
eatcatfish
eatcatfish

Reputation: 3

I have set all the header information, but I still can't get the data

I have used the browser to get the header information, and set all the header information, but still can't get the data normally.

    import requests
url='https://www.lagou.com/jobs/positionAjax.jsoncity=%E4%B8%8A%E6%B5%B7&needAddtionalResult=false'
headers={
'Host':'www.lagou.com',
'Connection':'keep-alive',
'Content-Length':'22',
'Pragma':'no-cache',
'Cache-Control':'no-cache',
'Origin':'https://www.lagou.com',
'X-Anit-Forge-Code':'0',
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8',
'Accept':'application/json, text/javascript, */*; q=0.01',
'X-Requested-With':'XMLHttpRequest',
'X-Anit-Forge-Token':'None',
'Referer':'https://www.lagou.com/jobs/list_php?labelWords=&fromSearch=true&suginput=',
'Accept-Encoding':'gzip,deflate,br',
'Accept-Language':'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7'
}
data={
    'first':'true',
    'pn':1,
    'kd':'php'
}
resp=requests.post(url=url,data=data,headers=headers)
print(resp.text)

Upvotes: 0

Views: 91

Answers (2)

jia Jimmy
jia Jimmy

Reputation: 1848

cookie in request header is also needed for pretending a normal web browser action:

you can keep these key field in your cookies params:


cookies = {
    'HMACCOUNT':'your_HMACCOUNT_value',
    'LGRID':'your_LGRID_value',
    'JSESSIONID':'your_JSESSIONID_value',
    'user_trace_token': 'your_user_trace_token_value',
    'PRE_LAND': 'your_PRE_LAND_value',
}

# all these value you can find in Browser Developer Mode, you know how to get it man.

then add cookies param in your request


resp=requests.post(url=url, data=data, cookies=cookies,  headers=headers)
print(resp.text)

here's total codes, wish it helpful to you:


import requests

url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false'
headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.9',
    'Connection': 'keep-alive',
    'Content-Length': '22',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'Host': 'www.lagou.com',
    'Origin': 'https://www.lagou.com',
    'Referer': 'https://www.lagou.com/jobs/list_PHP?px=new&city=%E5%8C%97%E4%BA%AC',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
    'X-Anit-Forge-Code': '0',
    'X-Anit-Forge-Token': 'None',
    'X-Requested-With': 'XMLHttpRequest',

}

data = {
    'first': 'true',
    'pn': 1,
    'kd': 'php'
}

cookies = {
    'HMACCOUNT': '04549EEF16D48382',
    'LGRID': '20190111154031-2c79703b-1574-11e9-990e-525400f775ce',
    'JSESSIONID': 'ABAAABAABEEAAJAC6C939E47B1C56C2C11F0557901B4343',
    'user_trace_token': '20190111141900-c94dbdcb-1568-11e9-9819-525400f775ce',
    'PRE_LAND': 'https%3A%2F%2Fpassport.lagou.com%2Flogin%2Flogin.html%3Fsignature%3DD82D1A9585AE6ED5D3AEA34E5CD1C9B0%26service%3Dhttp%25253A%25252F%25252Fwww.lagou.com%25252Fjobs%25252F%26action%3Dlogin%26serviceId%3Dlagou%26ts%3D1547187540500;',
}

resp = requests.post(url=url, data=data, cookies=cookies, headers=headers)
print(resp.text)

Upvotes: 1

NETcrypt0r
NETcrypt0r

Reputation: 1

From http://docs.python-requests.org/en/master/user/quickstart/:

I think the problem is with the way you have encoded or "formatted" your target URL(url='https://www.lagou.com/jobs/positionAjax.jsoncity=%E4%B8%8A%E6%B5%B7&needAddtionalResult=false') to obtain information from. The URL paramaters (variables) should be encoded by using a key value pair dictionary instead of an absolute URL path without properly formatted parameters. Ex:

payload = {'key1': 'value1', 'key2': ['value2', 'value3']}

r = requests.get('https://httpbin.org/get', params=payload)
print(r.url)
RESULT: https://httpbin.org/get?key1=value1&key2=value2&key2=value3

You often want to send some sort of data in the URL’s query string. If you were constructing the URL by hand, this data would be given as key/value pairs in the URL after a question mark, e.g. httpbin.org/get?key=val. Requests allows you to provide these arguments as a dictionary of strings, using the params keyword argument. As an example, if you wanted to pass key1=value1 and key2=value2 to httpbin.org/get, you would use the following code:

payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.get('https://httpbin.org/get', params=payload)

You should read the follow up over at: http://docs.python-requests.org/en/master/user/quickstart/ to gain some insight.

Upvotes: 0

Related Questions