CODEWITHSUNDEEP
pythonweb-scrapingpython-requestsstockquotes
gasper101
gasper101

Reputation: 122

Python, requests webscraping - NSE India gives emtpy list

I am trying to use requests to scrape data form NSE to find top gainers on Indian stock market. I have used this same method on yahoo finance and it worked but here, I keep getting the empty list as the result.

Here is my code:

import requests
from lxml import html
import urllib3


urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

resp = requests.get('https://www.nseindia.com/market-data/top-gainers-loosers',  verify=False, headers=headers)
tree = html.fromstring(resp.content)

count = 1
stocks = []
for i in range(30):
    name = tree.xpath('//*[@id="topgainer-Table"]/tbody/tr['+str(count)+']/td[1]/a')
    print(name)
    try:
        stocks.append(name[0].text)
    except:
        pass
    #name.text  
    count +=1
    
print(stocks)

As an output I get [ ] printed many times (an empty list). I think that the problem is with NSE because it has many different tables all with same xpaths.

Any ideas?

Upvotes: 0

Views: 2348

Answers (2)

Amit Ghosh
Amit Ghosh

Reputation: 1606

Another easier alternative, Use the nsefetch() function as documented here https://unofficed.com/nse-python/documentation/nsefetch/

In case you want python-requests method

from nsepython import *
payload= nsefetch('https://www.nseindia.com/market-data/top-gainers-loosers')
print(payload)

In case you want curl method

from nsepythonserver import *
payload= nsefetch('https://www.nseindia.com/market-data/top-gainers-loosers')
print(payload)

Upvotes: 0

Bhavya Parikh
Bhavya Parikh

Reputation: 3400

If you check the website it take some time to load data so from bs4 you will not be able to find data.

Network Tab::

enter image description here

You can find it from Network tab and search for any company name you can get from screenshot attached and find the URL and make call to it and it will return JSON data. so you can extract what so data you want

import requests
import pandas  as pd
headers={"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"}
res=requests.get(r"https://www.nseindia.com/api/live-analysis-variations?index=gainers",headers=headers)
all_data=res.json()['NIFTY']['data']
df=pd.DataFrame(all_data)

Output: enter image description here

URL for top loosers :: https://www.nseindia.com/api/live-analysis-variations?index=loosers

Upvotes: 3

Related Questions