CODEWITHSUNDEEP
python-3.xpandastime-seriestimestamp
Roosha
Roosha

Reputation: 65

How to read a very large csv file into a pandas dataframe as quickly as possible?

I am reading a very large csv file (~1 million rows) into a pandas dataframe using pd.read_csv() function according to the following options: (note that the seconds are also inside timestamp column but not shown in here due to exact copy and paste from csv file)

enter image description here

pd.read_csv(file,
            index_col='Timestamp',
            engine='c',
            na_filter=False,
            parse_dates=['Timestamp'],
            infer_datetime_format=True,
            low_memory=True)

My question is how to speed up the reading as it is taking forever to read the file?

Upvotes: 0

Views: 2981

Answers (0)

Related Questions