CODEWITHSUNDEEP
pythonloopsdataframeiterationpickle
Marlon Teixeira
Marlon Teixeira

Reputation: 393

How to save a list of dataframes while producing them iteratively?

I have a lot of folders, from which I am generating different dataframes, according to some criteria. As a result, the loop produces one dataframe each iteration. I want to save all of them in different .plk files. I'm just using it:

df.to_pickle('df')

But it only saves the last one. I need something like:

df.to_pickle(df[i].plk')

Where i is for each iteration.

Upvotes: 2

Views: 3062

Answers (3)

Marlon Teixeira
Marlon Teixeira

Reputation: 393

As a matter of contribution, I did it, and it has worked:

df[i].to_pickle(str(i)+'some_name.plk')

Upvotes: 0

Dev Khadka
Dev Khadka

Reputation: 5461

just replace your this line 

df.to_pickle(df[i].plk')

with 

df.to_pickle(f"df_{i}".plk)

it will save your files on df_1.plk, df_2.plk, df_3.plk etc

Upvotes: 2

Celius Stingher
Celius Stingher

Reputation: 18377

This is how I manage: First create an empty list where you will append each dataframe you create:

df_appender = []

Suppose I generate a list ids that contains unique values that will be used to iterate and generate each dataframe you desire:

ids = ['A','B','C','D','E']

Here is where the loop takes place and whatever logic you want to apply to create new dataframes and append them:

for i in range(len(ids)):
    dfx = df[df['id'] == ids[i]] 
    dfx = dfx.sort_values(['day'],ascending=[True])
    dfx = dfx.drop_duplicates(['day'],keep='last')
    df_appender.append(dfx)
    print(i/len(ids))

Now you have a list with dataframes for each iteration, finally if you want to concatenate them to one big dataframe you can do:

final_df = pd.concat(df_appender,ignore_index=True)

Upvotes: 4

Related Questions