CODEWITHSUNDEEP
pythonpandasdataframe
notArobot
notArobot

Reputation: 13

Multiple dataframes stored in dictionary - Python/Pandas

I have different csv and excel files to load in pandas and I was wondering to dynamically create the load structure (example: df1 = pd.read_csv(xxxxx)) so I'm looping thru the files and creating the command and storing them in a dictionary.

The problem the dictionary is storing the values as string and from there I can load the csv files and go thru the dataframe.

It's working if I use exec and eval but I'm looking for alternatives. so far: to create the command.

for i in list1:
     df_dic[calendar.month_abbr[int(i[4:6])]] = ''.join('''df_lst_'''+calendar.month_abbr[(int(i[4:6]))]+''' = pd.read_excel(r'''+"'"+dir1+i+"'"+','+'sheet_name='''''Data'''"'"+')')

then I'm creating a new dict only with the based only on the df_list[X] to manipulate it later but again it is been stored as a string. I've tried different approaches (ast.literal_eval for example, which is giving me ValueError: malformed node or string ). I'm stuck

Any ideas?

Appreciate any help.

Cheers

Upvotes: 1

Views: 8355

Answers (2)

Andy
Andy

Reputation: 450

You can use dict comprehensions:

import pandas as pd
paths = ['file1.csv', 'file2.csv']
dfs = {p: pd.read_csv(p) for p in paths}

If you also want to read Excel files is the same command, you can use

paths = ['file1.csv', 'file2.csv', 'excel_file.xls']
dfs = {p: pd.read_csv(p) if p.endswith('.csv') else pd.read_excel(p) for p in paths}

@jorge's solution with the loop has the advantage that you can wrap each read_csv in a try-catch block, so you can handle corrupt files.

Upvotes: 0

J L
J L

Reputation: 987

May you try this

def load_csvs(*paths):
    dfs = {}
    for path in paths:
        dfs[path] = pd.read_csv(path)
    return dfs

if __name__ == '__main__':
    paths = ['foo.csv', 'bar.csv']
    dfs = load_csvs(paths)
    # Access the foo.csv dataframe as foo_df
    foo_df = dfs['foo.csv']

You can access the dataframes via its paths to manipulate them and so on.

Upvotes: 2

Related Questions