CODEWITHSUNDEEP
pythonweb-scrapingpython-requests
NandaKishore.Rao
NandaKishore.Rao

Reputation: 131

Python requests 401 error but url opens in browser

I am trying to pull the json from this location - https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY

This opens fine in my browser, but using requests in python throws a 401 permission error. I have tried adding headers with different arguments, but to no avail. Interestingly, the json on this page does not open in the browser as well until https://www.nseindia.com is opened separately. I believe it requires some kind of authentication, but surprised it works in the browser without any.

Is there a way to extract the information from this url? Any help is much appreciated.

Here is my implementation -

import requests

url = 'https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY'

# This throws a 401 response error
page = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})

# This throws a 'Connection aborted' error
page = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (.NET CLR 3.5.30729)"})

Upvotes: 4

Views: 3410

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195438

To get the correct data, first load cookies from other URL with requests.get() and then do Ajax request to load the JSON:

import json
import requests
from bs4 import BeautifulSoup


headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'}
url = 'https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY'

with requests.session() as s:

    # load cookies:
    s.get('https://www.nseindia.com/get-quotes/derivatives?symbol=BANKNIFTY', headers=headers)

    # get data:
    data = s.get(url, headers=headers).json()

    # print data to screen:
    print(json.dumps(data, indent=4))

Prints:

{
    "records": {
        "expiryDates": [
            "03-Sep-2020",
            "10-Sep-2020",
            "17-Sep-2020",
            "24-Sep-2020",
            "01-Oct-2020",
            "08-Oct-2020",
            "15-Oct-2020",
            "22-Oct-2020",
            "29-Oct-2020",
            "26-Nov-2020"
        ],
        "data": [
            {
                "strikePrice": 18100,
                "expiryDate": "03-Sep-2020",
                "CE": {
                    "strikePrice": 18100,
                    "expiryDate": "03-Sep-2020",
                    "underlying": "BANKNIFTY",
                    "identifier": "OPTIDXBANKNIFTY03-09-2020CE18100.00",
                    "openInterest": 1,
                    "changeinOpenInterest": 1,
                    "pchangeinOpenInterest": 0,
                    "totalTradedVolume": 2,
                    "impliedVolatility": 95.51,
                    "lastPrice": 6523.6,
                    "change": 2850.1000000000004,
                    "pChange": 77.58540901048048,
                    "totalBuyQuantity": 2925,
                    "totalSellQuantity": 2800,
                    "bidQty": 25,
                    "bidprice": 6523.6,
                    "askQty": 25,
                    "askPrice": 6570.3,
                    "underlyingValue": 24523.8
                },
                "PE": {
                    "strikePrice": 18100,
                    "expiryDate": "03-Sep-2020",
                    "underlying": "BANKNIFTY",

...and so on.

Upvotes: 6

Related Questions