How to concatenate large dataset into dataframe pandas

Question

So I am working with a fairly substantial CSV dataset that has couple hundred megabytes. I have managed to read in the data in chunks (~100 rows). How do i then elegantly convert those chunks into a dataframe and apply the describe function to it? Thank you

Answer 1

It seems you need concat of TextFileReader object what is output of read_csv if parameter chunksize with describe:

df = pd.concat([x for x in pd.read_csv('filename', chunksize=1000)], ignore_index=True)
df = df.describe()
print (df)