CODEWITHSUNDEEP
pythonpandaslistdataframe
Raghavan vmvs
Raghavan vmvs

Reputation: 1265

Merge a list of list of dataframe python

I have a list of lists each containing a common column, A, B to be used as indices as follows

df_list=[[df1], [df2],[df3], [df4], [df5, df6]]

I would like to merge the dataframes into a single dataframe based on the common columns A, B in all dataframes

I have tried pd.concat(df_list). This doesnt work and yields an error

 list indices must be integers or slices, not list

Is there a way to accomplish this

Upvotes: 0

Views: 387

Answers (3)

Daweo
Daweo

Reputation: 36450

You need to deliver flat list (or other flat structure) to pd.concat your

df_list=[[df1], [df2],[df3], [df4], [df5, df6]]

is nested. If you do not have command over filling said df_list you need to flatten it first for example using itertools.chain as follow:

import itertools
flat_df_list = list(itertools.chain(*df_list))

and then deliver flat_df_list or harnessing fact that pd.concat also work with itertators:

total_df = pd.concat(itertools.chain(*df_list))

Upvotes: 4

Raghavan vmvs
Raghavan vmvs

Reputation: 1265

The solution is as follows

  flat_list = [item for sublist in df_list for item in sublist]
        
  #L3 = pd.DataFrame.from_dict(map(dict,D))
  L3=pd.concat(flat_list)

Upvotes: 1

snatchysquid
snatchysquid

Reputation: 1352

try simply using something like:

dfs = [df1, df2, df3, ... etc]

print(pd.concat(dfs))

when storing the dfs in a list you shouldn't keep them in a second list, note pd.concat takes a list of dfs.

Upvotes: 3

Related Questions