CODEWITHSUNDEEP
pythoncsvpandas
wolfsatthedoor
wolfsatthedoor

Reputation: 7303

Pandas Python: Writing output to specific cells in the CSV

I am trying to write a pandas df to a csv. However, I need to loop a lot to get my output in chunks. I want to stack the output vertically in a csv. So after each iteration, I want to write to a particular set of rows (1:10, then 11:20, etc...) and clear memory so as not to make a giant object. Is it possible to use df.to_csv to do this?

Upvotes: 3

Views: 712

Answers (1)

EdChum
EdChum

Reputation: 394051

to_csv accepts a mode argument to append:

import numpy as np
# df.shape returns the dimensions in a tuple, the first dimension is the number of rows
df_list = np.array_split(df, df.shape[0]/10)
for d in df_list:
    d.to_csv(csv_path, mode='a')

You can use numpy array split to produce a list of your df split into 10 rows each and then write them out or whatever you need to do to save memory

Upvotes: 5

Related Questions