CODEWITHSUNDEEP
pythonpandas
DyTech
DyTech

Reputation: 159

Making Dataframe Analysis faster

I am using three dataframes to analyze sequential numeric data - basically numeric data captured in time. There are 8 columns, and 360k entries. I created three identical dataframes - one is the raw data, the second a "scratch pad" for analysis and a third dataframe contains the analyzed outcome. This runs really slowly. I'm wondering if there are ways to make this analysis run faster? Would it be faster if instead of three separate 8 column dataframes I had one large one 24 column dataframe?

Upvotes: 0

Views: 111

Answers (3)

DyTech
DyTech

Reputation: 159

Rereading this post I am realizing I could have been clearer. I have been using write statement like: 

dm.iloc[p,XCol] = dh.iloc[x,XCol]

to transfer individual cells of one dataframe (dh) to a different row of a second dataframe (dm). It ran very slowly but I needed this specific file sorted and I just lived with the performance.

According to "Learning Pandas" by Michael Heydt, pg 146, ".iat" is faster than ".iloc" for extracting (or writing) scalar values from a dataframe. I tried it and it works. With my original 300k row files, run time was 13 hours(!) using ".iloc", same datafile using ".iat" ran in about 5 minutes.

Net - this is faster: dm.iat[p,XCol] = dh.iat[x,XCol]

Upvotes: 0

quant_dev
quant_dev

Reputation: 6231

Most probably it doesn't matter because pandas stores each column separately anyway (DataFrame is a collection of Series). But you might get better data locality (all data next to each other in memory) by using a single frame, so it's worth trying. Check this empirically.

Upvotes: 0

Mike Graham
Mike Graham

Reputation: 76683

Use cProfile and lineprof to figure out where the time is being spent.

To get help from others, post your real code and your real profile results.

Optimization is an empirical process. The little tips people have are often counterproductive.

Upvotes: 1

Related Questions