CODEWITHSUNDEEP
pythonpandasapply
piRSquared
piRSquared

Reputation: 294358

Why does pandas apply calculate twice

I'm using the apply method on a panda's DataFrame object. When my DataFrame has a single column, it appears that the applied function is being called twice. The questions are why? And, can I stop that behavior?

Code:

import pandas as pd

def mul2(x):
    print ('hello')
    return 2*x

df = pd.DataFrame({'a': [1,2,0.67,1.34]})
df.apply(mul2)

Output:

hello
hello

0  2.00
1  4.00
2  1.34
3  2.68

I'm printing 'hello' from within the function being applied. I know it's being applied twice because 'hello' printed twice. What's more is that if I had two columns, 'hello' prints 3 times. Even more still is when I call applied to just the column 'hello' prints 4 times.

Code:

df.a.apply(mul2)

Output:

hello
hello
hello
hello
0    2.00
1    4.00
2    1.34
3    2.68
Name: a, dtype: float64

Upvotes: 39

Views: 10330

Answers (3)

cs95
cs95

Reputation: 402673

This behavior has been fixed with pandas 1.1, please upgrade!

Now, apply and applymap on DataFrame evaluates first row/column only once.

Initially, we had GroupBy.apply and Series/df.apply evaluating the first group twice. The reason the first group is evaluated twice is because apply wants to know whether it can "optimize" the calculation (sometimes this is possible if apply receives a numpy or cythonized function). With pandas 0.25, this behavior was fixed for GroupBy.apply. Now, with pandas 1.1, this will also be fixed for df.apply.

Old Behavior [pandas <= 1.0.X]

pd.__version__ 
# '1.0.4'

df.apply(mul2)
hello
hello

      a
0  2.00
1  4.00
2  1.34
3  2.68

New Behavior [pandas >= 1.1]

pd.__version__
# '1.1.0.dev0+2004.g8d10bfb6f'

df.apply(mul2)
hello

      a
0  2.00
1  4.00
2  1.34
3  2.68

Upvotes: 10

MERose
MERose

Reputation: 4421

This behavior is intended, as an optimization.

See the docs:

In the current implementation apply calls func twice on the first column/row to decide whether it can take a fast or slow code path. This can lead to unexpected behavior if func has side-effects, as they will take effect twice for the first column/row.

Upvotes: 16

BrenBarn
BrenBarn

Reputation: 251408

Probably related to this issue. With groupby, the applied function is called one extra time to see if certain optimizations can be done. I'd guess something similar is going on here. It doesn't look like there's any way around it at the moment (although I could be wrong about the source of the behavior you're seeing). Is there a reason you need it to not do that extra call.

Also, calling it four times when you apply on the column is normal. When you get one columnm you get a Series, not a DataFrame. apply on a Series applies the function to each element. Since your column has four elements in it, the function is called four times.

Upvotes: 13

Related Questions