CODEWITHSUNDEEP
pythonpandasdictionarypathlib
user1442363
user1442363

Reputation: 850

How to create a dict of dataframes from multiple csv files

I am loading a csv file in pandas as

premier10 = pd.read_csv('./premier_league/pl_09_10.csv')

However, I have 20+ csv files, which I was hoping to load as separate dfs (one df per csv) using a loop and predefined names, something similar to:

import pandas as pd
file_names = ['pl_09_10.csv','pl_10_11.csv']
names = ['premier10','premier11']
for i in range (0,len(file_names)):
     names[i] = pd.read_csv('./premier_league/{}'.format(file_names[i]))

(Note, here I provide only two csv files as example) Unfortunately, this doesn't work (no error messages, but the the pd dfs don't exist).

Any tips/links to previous questions would be greatly appreciated as I haven't found anything similar on Stackoverflow.

Upvotes: 2

Views: 2203

Answers (3)

Trenton McKinney
Trenton McKinney

Reputation: 62573

  1. Use pathlib to set a Path, p, to the files
  2. Use the .glob method to find the files matching the pattern
  3. Create a dataframe with pandas.read_csv
    • Use a dict comprehension to create a dict of dataframes, where each file will have its own key-value pair.
      • Use the dict like any other dict; the keys are the file names and the values are the dataframes.
    • Alternatively, use a list comprehension with pandas.concat to create a single dataframe from all the files.
from pathlib import Path
import pandas as pd

# set the path to the files
p = Path('some_path/premier_league')  

# create a list of the files matching the pattern
files = list(p.glob(f'pl_*.csv'))

# creates a dict of dataframes, where each file has a separate dataframe
df_dict = {f.stem: pd.read_csv(f) for f in files}  

# alternative, creates 1 dataframe from all files
df = pd.concat([pd.read_csv(f) for f in files])

Upvotes: 2

Benjamin Parsons
Benjamin Parsons

Reputation: 35

This is what you want:

#create a variable and look through contents of the directory 
files=[f for f in os.listdir("./your_directory") if f.endswith('.csv')]

#Initalize an empty data frame
all_data = pd.DataFrame()

#iterate through files and their contents, then concatenate their data into the data frame initialized above
for file in files:
   df = pd.read_csv('./your_directory' + file)
   all_data = pd.concat([all_data, df])

#Call the new data frame and verify that contents were transferred
all_data.head()

Upvotes: 0

Gustave Coste
Gustave Coste

Reputation: 707

names = ['premier10','premier11'] does not create a dictionary but a list. Simply replace it with names = dict() or replace names = ['premier10','premier11'] by names.append(['premier10','premier11'])

Upvotes: 0

Related Questions