CODEWITHSUNDEEP
pythonajaxpython-requests
brized
brized

Reputation: 51

Python Requests AJAX Response Different from Browser Due to Cookie Handling

My request:

# python 3.7.3
import requests
from requests import Session

session = Session()
session.head('https://www.basspro.com/shop/en/blazer-brass-handgun-ammo')
cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies))
response = session.post(
    url='https://www.basspro.com/shop/BPSGetInventoryStatusByIDView',
    data={
        'productId': '3074457345616736172',
        'itemId': '3074457345616736949',
        'isGunFlag': 'false',
    },
    cookies=cookies,
    headers={
        'accept': '*/*',
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'en-US,en;q=0.9',
        'content-length': '72',
        'content-type': 'application/x-www-form-urlencoded',
        'origin': 'https://www.basspro.com',
        'referer': 'https://www.basspro.com/shop/en/blazer-brass-handgun-ammo',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.92 Safari/537.36 Vivaldi/2.9.1705.38',
        'x-requested-with': 'XMLHttpRequest',

    },
)

print(response.text)

Output:

<input type="hidden" class="relativeToAbsolute" value="true" />

/*
{
"onlineInventory": {
        "status": "Status Not Available",
        "image": "widget_product_info/outofstock_icon.svg",
        "altText": "Status Not Available",
        "isDropShip": false,
        "availableDate":""
},
"inStoreInventory": {
        "stores": [],
        "checkStoreText": "Check Store Availability",
        "isInStoreInventory": true,
        "isPickupInventory": false
}
}
*/

My output when inspecting and running the same AJAX request via browser: 

/*
{
"onlineInventory": {
    "status": "Backordered",
    "image": "widget_product_info/backordered_icon.svg",
    "altText": "Backordered",
    "isDropShip": false,    
    "quantity": 0,
    "availableDate":"May 1-8"
},
"inStoreInventory": {
    "stores": [{
                id: '715839555',
                name: '83',
                gunRestricted: 'false',
                dsName: 'TX - Round Rock',
                status: 'Unavailable',
                statusText: 'Out of Stock',
                image: 'widget_product_info/outofstock_icon.svg',
                altText: 'Out of Stock',
                availableDate: '',
                availableQuantity: '',
                availableQuantityDisplay: 'false',
                cityState: 'Round Rock, TX',
                ISPavailableDate: '',
                ISPavailableQuantity: '',
                pickupTime: 'by 2:00pm',
                offerISPOnBPS: 'Yes',
                offerISPOnCAB: 'No'}],
    "checkStoreText": "Change Store",
    "isInStoreInventory": true,
    "isPickupInventory": true       
}
}
*/

I tried assigning cookies this way as well:

url = "https://www.basspro.com/shop/en/blazer-brass-handgun-ammo"
r = requests.get(url)
cookies = r.cookies
# fails to pass the right cookie

If I instead copy the cookie verbatim from an inspected GET request at https://www.basspro.com/shop/en/blazer-brass-handgun-ammo and put that into the POST headers, it works. How do I get cookies to work properly programatically?

EDIT:

Here's my attempt at just using Session() for cookies:

# python 3.7.3
import requests
from requests import Session

session = Session()
session.get("https://www.basspro.com/shop/en/blazer-brass-handgun-ammo")
# session.head('https://www.basspro.com/shop/en/blazer-brass-handgun-ammo')
response = session.post(
    url='https://www.basspro.com/shop/BPSGetInventoryStatusByIDView',
    data={
        'productId': '3074457345616736172',
        'itemId': '3074457345616736949',
        'isGunFlag': 'false',
    },
    headers={
        'accept': '*/*',
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'en-US,en;q=0.9',
        'content-length': '72',
        'content-type': 'application/x-www-form-urlencoded',
        'origin': 'https://www.basspro.com',
        'referer': 'https://www.basspro.com/shop/en/blazer-brass-handgun-ammo',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.92 Safari/537.36 Vivaldi/2.9.1705.38',
        'x-requested-with': 'XMLHttpRequest',

    },
)

print(response.text)

I get the same result as before ("status": "Status Not Available", etc.)

Here's my attempt at the second solution:

# python 3.7.3
import requests
from requests import Session

url = "https://www.basspro.com/shop/en/blazer-brass-handgun-ammo"
r = requests.get(url)
cookies = r.cookies  # the type is RequestsCookieJar
response = requests.post(
    url='https://www.basspro.com/shop/BPSGetInventoryStatusByIDView',
    data={
        'productId': '3074457345616736172',
        'itemId': '3074457345616736949',
        'isGunFlag': 'false',
    },
    cookies=cookies,
    headers={
        'accept': '*/*',
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'en-US,en;q=0.9',
        'content-length': '72',

        'content-type': 'application/x-www-form-urlencoded',
        'origin': 'https://www.basspro.com',
        'referer': 'https://www.basspro.com/shop/en/blazer-brass-handgun-ammo',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.92 Safari/537.36 Vivaldi/2.9.1705.38',
        'x-requested-with': 'XMLHttpRequest',

    },
)

print(response.text)

Again, I get the same result as before. What am I doing wrong?

Upvotes: 1

Views: 324

Answers (1)

JianMing Wang
JianMing Wang

Reputation: 1169

can you try like this

session = Session()
session.get("https://www.basspro.com/shop/en/blazer-brass-handgun-ammo")

Then all the following calls with 

session.xxx

donot use cookies parameter in it

another way I have tested,

cookies = r.cookies  # the type is RequestsCookieJar
requests.post(.... cookies=cookies...)

at last ,I tested this works: Please compare carefully


from requests import Session

session = Session()
agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36'
r1 = session.get("https://www.basspro.com/shop/en/blazer-brass-handgun-ammo",headers={'user-agent': agent})
response = session.post(
    url='https://www.basspro.com/shop/BPSGetOnlineInventoryStatusByIDView',
    data={
        'productId': '3074457345616736172',
        'catalogId': '3074457345616676768',
        'storeId': '715838534',
        'langId':-1
    },
    headers={
        'user-agent': agent,
        'x-requested-with': 'XMLHttpRequest',
    },
    cookies=r1.cookies
)

print(response.text)

Upvotes: 1

Related Questions