CODEWITHSUNDEEP
pandas
scott martin
scott martin

Reputation: 1293

Pandas - Trying to save a set of files by reading it using Pandas but only the latest file gets saved

I am trying to read a set of txt files into Pandas as below. I see I am able to read them to a Dataframe however when I try to save the Dataframe it only saves the last file it read. However when I perform print(df) it prints all the records.

Given below is the code I am using:

files = '/users/user/files'
list = []
for file in files:
    df = pd.read_csv(file)
    list.append(df)
    print(df)
    df.to_csv('file_saved_path')

Could anyone advice why is the last file only being saved to the csv file and now the entire list.

Expected output:

output1
output2
output3

Current output:

output1,output2,output3

Upvotes: 0

Views: 30

Answers (2)

Loochie
Loochie

Reputation: 2472

Try this:

path = '/users/user/files'
for id in range(len(os.listdir(path))):
    file = os.listdir(path)[id]
    data = pd.read_csv(path+'/'+file, sep='\t')
    if id == 0:
        df1 = data
    else:
        data = pd.concat([df1, data], ignore_index=True)
data.to_csv('file_saved_path')

Upvotes: 1

jezrael
jezrael

Reputation: 862591

First change variable name list, because code word in python (builtin), then for final DataFrame use concat:

files = '/users/user/files'
L = []
for file in files:
    df = pd.read_csv(file)
    L.append(df)

bigdf = pd.concat(L, ignore_index=True)
bigdf.to_csv('file_saved_path')

Upvotes: 0

Related Questions