Pandas DataFrame use previous row value for complicated 'if' conditions to determine current value

Question

I want to know if there is any faster way to do the following loop? Maybe use apply or rolling apply function to realize this Basically, I need to access previous row's value to determine current cell value.

df.ix[0] = (np.abs(df.ix[0]) >= So) * np.sign(df.ix[0])
for i in range(1, len(df)):
    for col in list(df.columns.values):
        if ((df[col].ix[i] > 1.25) & (df[col].ix[i-1] == 0)) | :
            df[col].ix[i] = 1
        elif ((df[col].ix[i] < -1.25) & (df[col].ix[i-1] == 0)):
            df[col].ix[i] = -1
        elif ((df[col].ix[i] <= -0.75) & (df[col].ix[i-1] < 0)) | ((df[col].ix[i] >= 0.5) & (df[col].ix[i-1] > 0)):
            df[col].ix[i] = df[col].ix[i-1]
        else:
            df[col].ix[i] = 0

As you can see, in the function, I am updating the dataframe, I need to access the most updated previous row, so using shift will not work.

For example: Input:

A      B     C
1.3  -1.5   0.7
1.1  -1.4   0.6
1.0  -1.3   0.5
0.4   1.4   0.4

Output:

 A      B     C
1     -1      0
1     -1      0
1     -1      0
0      1      0

Answer 1

I am surprised there isn't a native pandas solution to this as well, because shift and rolling do not get it done. I have devised a way to do this using the standard pandas syntax but I am not sure if it performs any better than your loop... My purposes just required this for consistency (not speed).

import pandas as pd

df = pd.DataFrame({'a':[0,1,2], 'b':[0,10,20]})

new_col = 'c'

def apply_func_decorator(func):
    prev_row = {}
    def wrapper(curr_row, **kwargs):
        val = func(curr_row, prev_row)
        prev_row.update(curr_row)
        prev_row[new_col] = val
        return val
    return wrapper

@apply_func_decorator
def running_total(curr_row, prev_row):
    return curr_row['a'] + curr_row['b'] + prev_row.get('c', 0)

df[new_col] = df.apply(running_total, axis=1)

print(df)
# Output will be:
#    a   b   c
# 0  0   0   0
# 1  1  10  11
# 2  2  20  33

Disclaimer: I used pandas 0.16 but with only slight modification this will work for the latest versions too.

Others had similar questions and I posted this solution on those as well:

• Reference previous row when iterating through dataframe
• Reference values in the previous row with map or apply

Answer 2

@maxU has it right with shift, I think you can even compare dataframes directly, something like this:

df_prev = df.shift(-1)
df_out = pd.DataFrame(index=df.index,columns=df.columns)

df_out[(df>1.25) & (df_prev == 0)] = 1
df_out[(df<-1.25) & (df_prev == 0)] = 1
df_out[(df<-.75) & (df_prev <0)] = df_prev
df_out[(df>.5) & (df_prev >0)] = df_prev

The syntax may be off, but if you provide some test data I think this could work.

Saves you having to loop at all.

EDIT - Update based on comment below

I would try my absolute best not to loop through the DF itself. You're better off going column by column, sending to a list and doing the updating, then just importing back again. Something like this:

df.ix[0] = (np.abs(df.ix[0]) >= 1.25) * np.sign(df.ix[0]) 
for col in df.columns.tolist():
    currData = df[col].tolist()
    for currRow in range(1,len(currData)):
        if  currData[currRow]> 1.25 and currData[currRow-1]== 0:
            currData[currRow] = 1
        elif currData[currRow] < -1.25 and currData[currRow-1]== 0:
            currData[currRow] = -1
        elif currData[currRow] <=-.75 and currData[currRow-1]< 0:
            currData[currRow] = currData[currRow-1]
        elif currData[currRow]>= .5 and currData[currRow-1]> 0:
            currData[currRow] = currData[currRow-1]
        else:
            currData[currRow] = 0
    df[col] = currData

Answer 3

you can use .shift() function for accessing previous or next values:

previous value for col column:

df['col'].shift()

next value for col column:

df['col'].shift(-1)

Example:

In [38]: df
Out[38]:
   a  b  c
0  1  0  5
1  9  9  2
2  2  2  8
3  6  3  0
4  6  1  7

In [39]: df['prev_a'] = df['a'].shift()

In [40]: df
Out[40]:
   a  b  c  prev_a
0  1  0  5     NaN
1  9  9  2     1.0
2  2  2  8     9.0
3  6  3  0     2.0
4  6  1  7     6.0

In [43]: df['next_a'] = df['a'].shift(-1)

In [44]: df
Out[44]:
   a  b  c  prev_a  next_a
0  1  0  5     NaN     9.0
1  9  9  2     1.0     2.0
2  2  2  8     9.0     6.0
3  6  3  0     2.0     6.0
4  6  1  7     6.0     NaN