CODEWITHSUNDEEP
pythonpandaslist
Alexander Caskie
Alexander Caskie

Reputation: 357

Appending multiple pandas dataframe using for loop but returns an empty dataframe

I am to download a number of .csv files which I convert to pandas dataframe and append to each other.

The csv can be accessed via url which is created each day and using datetime it can be easily generated and put in a list.

I am able to open these individually in the list.

When I try to open a number of these and append them together I get an empty dataframe. The code looks like this so.

#Imports
import datetime 
import pandas as pd


#Testing can open .csv file
data = pd.read_csv('https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin01022018.csv')
data.iloc[:5]


#Taking heading to use to create new dataframe 
data_headings = list(data.columns.values)


#Setting up string for url
path_start = 'https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin'
file = ".csv"


#Getting dates which are used in url 
start = datetime.datetime.strptime("01-02-2018", "%d-%m-%Y")
end = datetime.datetime.strptime("04-02-2018", "%d-%m-%Y")
date_generated = [start + datetime.timedelta(days=x) for x in range(0, (end-start).days)]


#Creating new dataframe which is appended to
for heading in data_headings:
    data = {heading: []}

df = pd.DataFrame(data, columns=data_headings)


#Creating list of url
date_list = []

for date in date_generated:
    date_string = date.strftime("%d%m%Y")

    x = path_start + date_string + file
    date_list.append(x)


#Opening and appending csv files from list which contains url
for full_path in date_list:
    data_link = pd.read_csv(full_path)
    df.append(data_link)

print(df)

I have checked that they are not just empty csv but they are not. Any help would be appreciated.

Cheers, Sandy

Upvotes: 2

Views: 1581

Answers (2)

Cilantro Ditrek
Cilantro Ditrek

Reputation: 1097

You are never storing the appended dataframe. The line:

df.append(data_link)

Should be

df = df.append(data_link)

However, this may be the wrong approach. You really want to use the array of URLs and concatenate them. Check out this similar question and see if it can improve your code!

Upvotes: 3

Lucas Hort
Lucas Hort

Reputation: 854

I really can't understand what you wanted to do here:

#Creating new dataframe which is appended to
for heading in data_headings:
    data = {heading: []}

df = pd.DataFrame(data, columns=data_headings)

By the way, try this:

for full_path in date_list:
    data_link = pd.read_csv(full_path)
    df.append(data_link.copy())

Upvotes: 0

Related Questions