CODEWITHSUNDEEP
javascriptpythonhtmlweb-scraping
yash
yash

Reputation: 27

How to download data behind buttons which call javascript functions?

I was trying to download this data from the website. https://www.nseindia.com/market-data/oi-spurts

enter image description here

How can scrape this using python?

Upvotes: 1

Views: 437

Answers (2)

eliottness
eliottness

Reputation: 135

One way to find the final download link to a certain file is to open the debugger of your web browser and to click on the download link while looking into the Networking tab of the debugger.

Normally you will see the request the javascript of the page called as same as the url, the content of the request and so on...

From here you just need to replicate what request was sent by the javascript.

Upvotes: 0

Paul M.
Paul M.

Reputation: 10809

The JavaScript function downloadCSV is part of gijgo.min.js. It invokes getCSV, which goes through the fields in the table and generates a CSV file on the fly.

Fortunately, you don't have to deal with CSVs or scraping anything from the page. To get the data you want, all you have to do is make an HTTP GET request to the same RESTful API that your browser makes a request to when visiting the page:

def main():

    import requests

    url = "https://www.nseindia.com/api/live-analysis-oi-spurts-underlyings"

    headers = {
        "user-agent": "Mozilla/5.0"
    }

    response = requests.get(url, headers=headers)
    response.raise_for_status()

    data = response.json()["data"]

    print(f"There are {len(data)} items in total.")

    print(f"The first item is:\n{data[0]}")
    
    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

Output:

There are 143 items in total.
The first item is:
{'symbol': 'MOTHERSUMI', 'latestOI': 7182, 'prevOI': 4674, 'changeInOI': 2508, 'avgInOI': 53.66, 'volume': 12519, 'futValue': 53892.6066, 'optValue': 3788085280, 'total': 55585.0344, 'premValue': 1692.4278, 'underlyingValue': 104}
>>>

Upvotes: 1

Related Questions