CODEWITHSUNDEEP
pythonfor-looppandasdataframe
jonas
jonas

Reputation: 13969

Pandas, build new dataframe with for loop

I have a really simple problem that I cant solve in Pandas. I have a dataframe to start with, with that dataframe I want to apply some function. I want to repeat this many times and build/stack the reults from the operations in a new larger dataframe. I was thinking of doing this with a for loop. Here is a simplified example that I can not get to work:

import pandas as pd

df = pd.DataFrame(np.random.randn(3, 4), columns=list('ABCD'))

large_df = df*0

for i in range(1,10):
    df_new = df*i
    large_df= pd.concat(large_df,df_new)

large_df

Any ideas??

Upvotes: 4

Views: 5195

Answers (1)

Dan Allan
Dan Allan

Reputation: 35235

It will be fastest to build all of the results first and concatenate once in the end. If you append one result at a time, the memory for the results has to be re-allocated each time.

So, if you are applying some_function with a different parameter p through the loop (like i in your toy example above) I suggest:

pd.concat([df.apply(lambda x: some_function(x, p)) for p in parameters])

Upvotes: 5

Related Questions