CODEWITHSUNDEEP
pythonhtmlbeautifulsouppython-requests
otaku
otaku

Reputation: 1029

Href attribute not showing for <a> tag when using requests.post

I am trying to download the .csv file that appears on this page when I submit the data as

Data for : Security-wise Price volume & Deliverable position data
Symbol : 3INFOTECH
Select Series : All
Period : 24 months

My code is 

symbol = "3IINFOTECH"
url = "https://www.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp"
data = {

    "dataType":"priceVolumeDeliverable",
    "symbol":symbol,
    "segmentLink":"3",
    "symbolCount":"2",
    "series":"ALL",
    "rdPeriod":"groupPeriod",
    "dateRange":"24month"
}

headers = {
   'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11'
}
print("fetching for " + symbol)
session = requests.session()
response = requests.post(url, data, headers = headers)
html_content = response.text

soup = BeautifulSoup(html_content, "html.parser")

download_link = soup.findAll("span", attrs = {"class":"download-data-link"})[0]
print(download_link.a["href"])

Now on inspecting element I see this
enter image description here

How do I download the csv file? The post request from my code does not show me the href attribute.

enter image description here

Upvotes: 2

Views: 153

Answers (1)

Padraic Cunningham
Padraic Cunningham

Reputation: 180461

To get the link you have to click the button, so you could use selenium or something equivalent but it is very easy to just parse the data yourself as all you get back from the post request is the data:

symbol = "3IINFOTECH"
url = "https://www.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp"
data = {

    "dataType": "priceVolumeDeliverable",
    "symbol": symbol,
    "segmentLink": "3",
    "symbolCount": "2",
    "series": "ALL",
    "rdPeriod": "groupPeriod",
    "dateRange": "24month"
}

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11'
}
print("fetching for " + symbol)
import csv
response = requests.post(url, data, headers=headers)
html_content = response.text
soup = BeautifulSoup(html_content, "html.parser")
cols = [th.text for th in soup.select("th")]

rows = ([td.text for td in row.select("td")] for row in soup.select("tr  + tr"))

with open("data.csv", "w") as f:
    wr = csv.writer(f)
    wr.writerow(cols)
    wr.writerows(rows)

A snippet of data.csv:

Symbol,Series,Date,Prev Close,Open Price,High Price,Low Price,Last Price,Close Price,VWAP,Total Traded Quantity,Turnover in Lacs,No. of Trades,DeliverableQty,% Dly Qt toTraded Qty
3IINFOTECH,EQ,23-May-2014,9.90,10.25,10.70,9.70,10.10,10.10,10.23,"84,99,408",869.20,"16,539","40,35,648",47.48
3IINFOTECH,EQ,26-May-2014,10.10,10.40,10.60,9.10,9.30,9.20,9.97,"59,15,990",589.88,"9,894","27,10,021",45.81
3IINFOTECH,EQ,27-May-2014,9.20,9.20,9.30,8.30,8.60,8.55,8.53,"34,95,072",298.18,"3,600","14,71,141",42.09
3IINFOTECH,EQ,28-May-2014,8.55,8.60,9.40,8.45,9.30,9.15,9.07,"36,09,261",327.27,"3,955","13,92,733",38.59
3IINFOTECH,EQ,29-May-2014,9.15,9.25,9.50,8.80,9.40,9.35,9.28,"30,13,036",279.69,"3,090","15,20,654",50.47
3IINFOTECH,EQ,30-May-2014,9.35,9.35,9.55,8.90,9.00,9.00,9.13,"13,97,140",127.53,"1,992","7,43,964",53.25

Upvotes: 1

Related Questions