CODEWITHSUNDEEP
pythoncsvpandas
CreamStat
CreamStat

Reputation: 2185

Concatenate multiple similar CSV files into one big dataframe

I have one directory where there are only the CSV files I want to use. I want to concatenate all these CSV files and create a bigger one. I've tried one code but it didn't work.

import os
import pandas as pd

targetdir = r'C:/Users/toshiba/Documents/ICF2011/Base Admision San Marcos 2014-2/Sabado'

filelist = os.listdir(targetdir) 

big_df=pd.DataFrame()
for file in filelist :
    big_df.append(pd.read_csv(file), ignore_index=True)

I run the code and a there's a message that says: IOError: File A011.csv does not exist. This is contradictory because A011.csv is the first CSV file in the directory that I used.

Upvotes: 2

Views: 238

Answers (2)

Andy Hayden
Andy Hayden

Reputation: 375475

As mentioned in the other answer, you need to use the full-path, rather than the local path.

I recommend using concat rather than append, as this way you don't make many intermediary frames:

big_df = pd.concat(pd.read_csv(os.path.join(targetdir, filename),
                               ignore_index=True)
                   for filename in filelist)

Upvotes: 2

chthonicdaemon
chthonicdaemon

Reputation: 19760

listdir only returns the filename, not the complete path. To get the complete path you will need to join targetdir and file (bad variable name as it masks the file type). Also, you will have to capture the result of .append as it returns a new object rather than appending in place.

for filename in filelist:
    big_df = big_df.append(pd.read_csv(os.path.join(targetdir, filename), ignore_index=True)

Upvotes: 2

Related Questions