Calculate standard deviation for some rows in pandas dataframe that satisfy condition

Question

I would like to calculate standard deviation of 'values' in pandas dataframe under condition that it is calculated for common 'grouped_measurement'. After calculating I would like to calculate commented lines.

I tried following line:

df['standard_deviation'] = df['grouped_measurement'].diff().fillna(df['value']).std()

but it isn't working as expected. My full code is shown below:

import pandas as pd 
import numpy as np

# Define input dataframe
df = {'servo_in_position':      [1,1,1,0,0,0,1,1,1,1,1,0,0,0,0,1,1,1,1,1,1],
        'value':                [0.2,2.1,3.5,6.7,2.1,3.4,5.7,9.6,3.2,1.2,6.3,8.5,7.4,6.2,3.4,3.8,1.7,2.8,7.6,4.5,9.0]}

df = pd.DataFrame(df,columns= ['servo_in_position','value'])
print("Dataframe is:\n",df)

print("Grouping data according to servo positions, please wait...")
df['grouped_measurement'] = df['servo_in_position'].diff().fillna(df['servo_in_position']).eq(1).cumsum().mask(df['servo_in_position'] == 0, 0)
df['standard_deviation'] = df['grouped_measurement'].diff().fillna(df['value']).std()

# df=df.groupby('grouped_measurement',as_index=False).mean()
# df['new_value']=df['standard_deviation']*100/df['value']

print("Data grouped successfully!")
print("Input data:\n",df)

Expected output is following:

       servo_in_position       value        grouped_measurement      standard_deviation
0                1                0.2                1                1.6563011
1                1                2.1                1                1.6563011
2                1                3.5                1                1.6563011
3                0                6.7                0                0
4                0                2.1                0                0
5                0                3.4                0                0
6                1                5.7                2                3.194526569
7                1                9.6                2                3.194526569
8                1                3.2                2                3.194526569
9                1                1.2                2                3.194526569
10                1                6.3                2                3.194526569
11                0                8.5                0                0
12                0                7.4                0                0
13                0                6.2                0                0
14                0                3.4                0                0
15                1                3.8                3                2.832666588
16                1                1.7                3                2.832666588
17                1                2.8                3                2.832666588
18                1                7.6                3                2.832666588
19                1                4.5                3                2.832666588
20                1                9                 3                2.832666588

Answer 1

You can simplify your code - create Series s1 and s2 and for second use GroupBy.transform with std for filling new column with aggregtae values, also added numpy.where for set 0 by condition:

mask = df['servo_in_position'] == 0
s1 = df['servo_in_position'].diff().ne(0).cumsum()
s2 = df['value'].groupby(s1).transform('std')

#if need omit helper column simple remove df['grouped_measurement'] = np.where(mask, 0, s1)
df['grouped_measurement'] = np.where(mask, 0, s1)
df['standard_deviation'] = np.where(mask, 0, s2)
print("Dataframe is:\n",df)

     servo_in_position  value  grouped_measurement  standard_deviation
0                   1    0.2                     1            1.656301
1                   1    2.1                     1            1.656301
2                   1    3.5                     1            1.656301
3                   0    6.7                     0            0.000000
4                   0    2.1                     0            0.000000
5                   0    3.4                     0            0.000000
6                   1    5.7                     3            3.194527
7                   1    9.6                     3            3.194527
8                   1    3.2                     3            3.194527
9                   1    1.2                     3            3.194527
10                  1    6.3                     3            3.194527
11                  0    8.5                     0            0.000000
12                  0    7.4                     0            0.000000
13                  0    6.2                     0            0.000000
14                  0    3.4                     0            0.000000
15                  1    3.8                     5            2.832667
16                  1    1.7                     5            2.832667
17                  1    2.8                     5            2.832667
18                  1    7.6                     5            2.832667
19                  1    4.5                     5            2.832667
20                  1    9.0                     5            2.832667

Answer 2

First we create a series s which defines each change of servo_in_pisition as an unique group.

Then we GroupBy.transform(std) on these groups. We use transform to get an equal length vector back so we can define it as a new column to our exisiting dataframe, else the data would get aggregated.

We use np.where to conditionally assign the std if the value of servo_in_pisition != 0

s1 = df['servo_in_position'].diff().ne(0).cumsum()
s2 = df.groupby(s)['value'].transform('std')

df['standard_deviation'] = np.where(df['servo_in_position'].ne(0), s2, 0)

Output

    servo_in_position  value  standard_deviation
0                   1    0.2            1.656301
1                   1    2.1            1.656301
2                   1    3.5            1.656301
3                   0    6.7            0.000000
4                   0    2.1            0.000000
5                   0    3.4            0.000000
6                   1    5.7            3.194527
7                   1    9.6            3.194527
8                   1    3.2            3.194527
9                   1    1.2            3.194527
10                  1    6.3            3.194527
11                  0    8.5            0.000000
12                  0    7.4            0.000000
13                  0    6.2            0.000000
14                  0    3.4            0.000000
15                  1    3.8            2.832667
16                  1    1.7            2.832667
17                  1    2.8            2.832667
18                  1    7.6            2.832667
19                  1    4.5            2.832667
20                  1    9.0            2.832667