CODEWITHSUNDEEP
pythonpandasdataframe
Kevin Nash
Kevin Nash

Reputation: 1561

Pandas - Breaking a huge Dataframe into smaller chunks

I have a Dataframe of a million rows. I am trying to break it up into small sized Dataframes of 1000 rows each. I am able to break this huge Dataframe into smaller chunks (of 1000 rows each) using the below code:

size = 1000
list_of_dfs = [df[i:i+size-1,:] for i in range(0, len(df),size)]

I am however unsure how to call each of these smaller Dataframes in a loop so that I can read the entire 1000 small Dataframes that have been created

Upvotes: 1

Views: 773

Answers (1)

jezrael
jezrael

Reputation: 862441

Output is list of DataFrames, so you can loop them:

out = []
for df_small in list_of_dfs:
    print (df_small)
    #procesing...
    out.append(df_small)

Similar:

for i in range(0, len(df),size):
    df_small = df[i:i+size-1,:] 
    print (df_small)

If data are in csv is possible use chunksize parameter:

for df_small in pd.read_csv(filename, chunksize=size):
    print (df_small)

Upvotes: 3

Related Questions