CODEWITHSUNDEEP
pythoncsv
Ravi Shankaran
Ravi Shankaran

Reputation: 35

How to download multiple CSV files off a website using Python?

I am a beginner at programming (Finance professional) & I am looking to cut manual work using Python. I want to download multiple CSVs (Daily Volatility CSVs of past one year) from https://www.nseindia.com/products/content/equities/equities/archieve_eq.htm

So far, I am able to download one file at a time. But I am not able to apply for loop to download past one year's CSVs. Also, it would help if I can skip downloading CSVs from Saturdays and Sundays.

I made a csv file where links to all required CSV files are mentioned. Then tried to import that csv file and run a for loop operation on it. But I don't know enough programming to do that.

import requests
import shutil

r = requests.get('https://nseindia.com/archives/nsccl/volt/CMVOLT_01072018.CSV', stream=True)
if r.status_code == 200:
    with open("01072018.csv", 'wb') as f:
        r.raw.decode_content = True
        shutil.copyfileobj(r.raw, f)

Desired results: Download CSV files based on a date range input.
Actual results: Downloading 1 CSV file at a time.

Upvotes: 1

Views: 2958

Answers (3)

rohit arora
rohit arora

Reputation: 51

Alright without adding another library, the following is the code which should work even though it didn't work on my machine which has some restrictions.

import datetime as timer
import requests
import shutil


def download_data(date):
    url='https://nseindia.com/archives/nsccl/volt/CMVOLT_'+date+'.CSV'
    csv_filename=date+'.csv'
    try:
        print('Calling url:- ' + url)
        r = requests.get(url, stream=True,verify=False)
        if r.status_code == 200:
            with open(csv_filename, 'wb') as f:
                r.raw.decode_content = True
                shutil.copyfileobj(r.raw, f)
        r.close()
    except Exception as e:
        print('for Date '+ date +' Exception happened, most probably a weekend, EXCEPTION Message is ' + str(e))



def code_runner():
    i=0
    now = timer.datetime.now()
    day = now.day
    month = now.month
    year = now.year
    while i<365:
        day=day-1
        if day==0:
            day=31
            month=month-1
            if month==0:
                month=12
                year=year-1
        year1=year
        month1='{:02d}'.format(month)
        day1='{:02d}'.format(day)
        date=str(day1)+str(month1)+str(year1)
        download_data(date)
        i+=1

if __name__=='__main__':
    code_runner()

Upvotes: 1

msi_gerva
msi_gerva

Reputation: 2078

I would add a date loop for your script:

#!/usr/bin/env ipython
# --------------------
import requests
import shutil
import datetime
# -----------------------------------------------------------------------------------
dates=[datetime.datetime(2019,1,1)+datetime.timedelta(dval) for dval in range(0,366)];
# -----------------------------------------------------------------------------------
for dateval in dates:
    r = requests.get('https://www.nseindia.com/archives/nsccl/volt/CMVOLT_'+dateval.strftime('%d%m%Y')+'.CSV', stream=True)
    if r.status_code == 200:
        with open(dateval.strftime('%d%m%Y')+".csv", 'wb') as f:
            r.raw.decode_content = True
            shutil.copyfileobj(r.raw, f)
    # ---------------------------------------------------------------------------------

Upvotes: 0

McDonalds Happy Meal
McDonalds Happy Meal

Reputation: 23

filenames=['https://nseindia.com/archives/nsccl/volt/CMVOLT_01072018.CSV',
'https://nseindia.com/archives/nsccl/volt/CMVOLT_01082018.CSV',
'https://nseindia.com/archives/nsccl/volt/CMVOLT_01092018.CSV',
]

for x in filenames:
    r=requests.get(x, stream=True)
    if r.status_code == 200:
        with open(x.split('_')[-1], 'wb') as f:
            r.raw.decode_content = True
            shutil.copyfileobj(r.raw, f)

Upvotes: 1

Related Questions