CODEWITHSUNDEEP
google-apps-scriptweb-scrapingautomationrequest
Paresh Nagore
Paresh Nagore

Reputation: 486

Access nse option chain data (public API) using google app script

I want to access following url: https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY

  1. When I directly paste this in browser, it works like charm and gives an JSON object. (try it)

  2. Then I tried it with python using jupyter notebook. It worked again. Following code gives same object as in method 1 above.

    import requests
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36','Accept-Encoding': 'gzip, deflate, br','Accept-Language': 'en-US,en;q=0.9,hi;q=0.8'}
    url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
    json_obj = requests.get(nurl, headers = headers).json()
  1. But when I try accessing it with google app script, I get nothing. It keeps executing request and then timeout occurs after 6 minutes. Below is code for same
    var ocurl = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY";
  
    var hds = {
      'Accept-Encoding': 'gzip, deflate, br',
      'Accept-Language': 'en-US,en;q=0.9,hi;q=0.8',
      'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'
    };

    var params = {headers: hds, followRedirects: false, muteHttpExceptions: true};
    var ocresponse = UrlFetchApp.fetch(ocurl, params);
    //var data = JSON.parse(ocresponse.getContentText())

    Logger.log(ocresponse.getContentText());

My guess is that in first two cases request is going from local browser/server, but in case of third request it is going from servers owned by google. And NSE is blocking those requests. I'm not sure whether this is correct.

I want to use google script because it provides automatic trigger functionality (similar to cron job) and it is free.

Is there any way I can access it with google app script??

Upvotes: 3

Views: 21083

Answers (2)

CodingIsFun
CodingIsFun

Reputation: 108

I used the following headers and it worked

"Accept", "[asterisk]/[asterisk]"

"Accept-Encoding", "gzip, deflate, br"

"User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

"Connection", "keep-alive"

Upvotes: 2

Tanaike
Tanaike

Reputation: 201553

When you want to convert the following python script to Google Apps Script,

import requests
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36','Accept-Encoding': 'gzip, deflate, br','Accept-Language': 'en-US,en;q=0.9,hi;q=0.8'}
url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
json_obj = requests.get(nurl, headers = headers).json()

it becomes as follows.

var ocurl = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY";
var hds = {
  'Accept-Encoding': 'gzip, deflate, br',
  'Accept-Language': 'en-US,en;q=0.9,hi;q=0.8',
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'
};
var params = {headers: hds, muteHttpExceptions: true};
var ocresponse = UrlFetchApp.fetch(ocurl, params);

But, unfortunately, in the current stage, UrlFetchApp cannot modify User-Agent. By this, when above Google Apps Script is run, the value of User-Agent is Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa) where aaa... seems the unique ID. About modifying User-Agent using UrlFetchApp, it has already been reported at Google issue tracker as the future request. Ref Unfortunately, it seems that this has still not been able to be implemented.

From your replying in comments, it seems that User-Agent is required to be modified. So, unfortunately, as the current answer, it seems that your goal cannot be achieved using UrlFetchApp.

Reference:

Upvotes: 4

Related Questions