CODEWITHSUNDEEP
pythonpython-requests
Noobie Webscraping
Noobie Webscraping

Reputation: 3

Python request gives JSONDecodeError

I have the following curl statement that is providing a response in json format:

curl 'https://www.accenture.com/nl-en/searchbykeywords.search' \
 -H 'authority: www.accenture.com' \
 -H 'accept: application/json, text/javascript, */*; q=0.01' \
 -H 'dnt: 1' \
 -H 'x-requested-with: XMLHttpRequest' \
 -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36' \
 -H 'content-type: application/json; charset=UTF-8' \
 -H 'origin: https://www.accenture.com' \
 -H 'sec-fetch-site: same-origin' \
 -H 'sec-fetch-mode: cors' \
 -H 'sec-fetch-dest: empty' \
 -H 'referer: https://www.accenture.com/nl-en/search/results?srk=covid&pg=1&sb=0&filter=' \
 -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
 -H 'cookie: eVar46=covid' \
 --data-binary '{"k":"covid","f":1,"s":10, "sb":0, "ss":"" ,"cs":"true"}' \
 --compressed

But when I'm trying to replicate this via Python with help of the requests module I'm not receiving back the json which is causing the decode error. 

import requests

with requests.Session() as session:

    header = {
        'authority': 'www.accenture.com', 
        'accept': 'application/json, text/javascript, */*; q=0.01' ,
        'dnt': '1',
        'x-requested-with': 'XMLHttpRequest',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36',
        'content-type': 'application/json; charset=UTF-8' ,
        'origin': 'https://www.accenture.com' ,
        'sec-fetch-site': 'same-origin', 
        'sec-fetch-mode': 'cors' ,
        'sec-fetch-dest': 'empty',
        'referer': 'https://www.accenture.com/nl-en/search/results?srk=covid&pg=1&sb=0&filter=', 
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
        'cookie': 'eVar46=covid'
    }

    data = '{"k":"COVID","f":1,"s":10, "sb":0, "ss":"", "cs":"true"}'

    search_url = 'https://www.accenture.com/nl-en/searchbykeywords.search'

    r = session.post(search_url, headers=header, data=data)

    data = r.json()

    print(data)

Can you please help me out since the available answers didn't provide me with a working solution . Thanks

Upvotes: 0

Views: 106

Answers (1)

buran
buran

Reputation: 14273

import requests

with requests.Session() as session:
    header = {
        'authority': 'www.accenture.com', 
        'accept': 'application/json, text/javascript, */*; q=0.01' ,
        'dnt': '1',
        'x-requested-with': 'XMLHttpRequest',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36',
        'content-type': 'application/json; charset=UTF-8' ,
        'origin': 'https://www.accenture.com' ,
        'sec-fetch-site': 'same-origin', 
        'sec-fetch-mode': 'cors' ,
        'sec-fetch-dest': 'empty',
        'referer': 'https://www.accenture.com/nl-en/search/results?srk=covid&pg=1&sb=0&filter=', 
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
        'cookie': 'eVar46=covid'
    }

    data = {"k":"COVID","f":1,"s":10, "sb":0, "ss":"", "cs":"true"}
    search_url = 'https://www.accenture.com/nl-en/searchbykeywords.search'
    r = session.post(search_url, headers=header, json=data)
    data = r.json()
    print(data)

you can read more here - More complicated POST requests

Upvotes: 1

Related Questions