CODEWITHSUNDEEP
pythonpandasdataframetime-series
Fabio Lamanna
Fabio Lamanna

Reputation: 21574

pandas - groupby and filtering for consecutive values

I have this dataframe df:

U,Datetime
01,2015-01-01 20:00:00
01,2015-02-01 20:05:00
01,2015-04-01 21:00:00
01,2015-05-01 22:00:00
01,2015-07-01 22:05:00
02,2015-08-01 20:00:00
02,2015-09-01 21:00:00
02,2014-01-01 23:00:00
02,2014-02-01 22:05:00
02,2015-01-01 20:00:00
02,2014-03-01 21:00:00
03,2015-10-01 20:00:00
03,2015-11-01 21:00:00
03,2015-12-01 23:00:00
03,2015-01-01 22:05:00
03,2015-02-01 20:00:00
03,2015-05-01 21:00:00
03,2014-01-01 20:00:00
03,2014-02-01 21:00:00

made by U and a Datetime object. What I would like to do is to filter U values having at least three consecutive occurrences in months/year. So far I have grouped by by U, year and month as:

m = df.groupby(['U',df.index.year,df.index.month]).size()

obtaining:

U          
1  2015  1     1
         2     1
         4     1
         5     1
         7     1
2  2014  1     1
         2     1
         3     1
   2015  1     1
         8     1
         9     1
3  2014  1     1
         2     1
   2015  1     1
         2     1
         5     1
         10    1
         11    1
         12    1

The third column is related to the occurrences in different months/year. In this case only U values of 02 and 03 contain at least three consecutive values in months/year. Now I can't figured out how can I select those users and getting them out in a list, for instance, or just keeping them in the original dataframe df and discard the others. I tried also:

g = m.groupby(level=[0,1]).diff()

But I can't get any useful information.

Upvotes: 8

Views: 3098

Answers (1)

Nader Hisham
Nader Hisham

Reputation: 5414

Finally I could come up with the solution :) .

to give you an idea of how custom function works , simply it subtracts the value of the month from it's preceding value , the result should be one of course , and this should happen twice , for example if you have a list of numbers [5 , 6 , 7] , so 7 - 6 = 1 and 6 - 5 = 1 , 1 here appeared twice so the condition has been fulfilled 

In [80]:
df.reset_index(inplace=True)

In [281]:
df['month'] = df.Datetime.dt.month
df['year'] = df.Datetime.dt.year
df
Out[281]:
            Datetime    U   month   year
0   2015-01-01 20:00:00 1   1       2015
1   2015-02-01 20:05:00 1   2       2015
2   2015-04-01 21:00:00 1   4       2015
3   2015-05-01 22:00:00 1   5       2015
4   2015-07-01 22:05:00 1   7       2015
5   2015-08-01 20:00:00 2   8       2015
6   2015-09-01 21:00:00 2   9       2015
7   2014-01-01 23:00:00 2   1       2014
8   2014-02-01 22:05:00 2   2       2014
9   2015-01-01 20:00:00 2   1       2015
10  2014-03-01 21:00:00 2   3       2014
11  2015-10-01 20:00:00 3   10      2015
12  2015-11-01 21:00:00 3   11      2015
13  2015-12-01 23:00:00 3   12      2015
14  2015-01-01 22:05:00 3   1       2015
15  2015-02-01 20:00:00 3   2       2015
16  2015-05-01 21:00:00 3   5       2015
17  2014-01-01 20:00:00 3   1       2014
18  2014-02-01 21:00:00 3   2       2014

In [284]:
g = df.groupby([df['U'] , df.year])

In [86]:
res = g.filter(lambda x : is_at_least_three_consec(x['month'].diff().values.tolist()))
res
Out[86]:
      Datetime          U   month   year
7   2014-01-01 23:00:00 2   1       2014
8   2014-02-01 22:05:00 2   2       2014
10  2014-03-01 21:00:00 2   3       2014
11  2015-10-01 20:00:00 3   10      2015
12  2015-11-01 21:00:00 3   11      2015
13  2015-12-01 23:00:00 3   12      2015
14  2015-01-01 22:05:00 3   1       2015
15  2015-02-01 20:00:00 3   2       2015
16  2015-05-01 21:00:00 3   5       2015

if you want to see the result of the custom function

In [84]:
res = g['month'].agg(lambda x : is_at_least_three_consec(x.diff().values.tolist()))
res
Out[84]:
U  year
1  2015    False
2  2014     True
   2015    False
3  2014    False
   2015     True
Name: month, dtype: bool

this is how custom function implemented

In [53]:    
def is_at_least_three_consec(month_diff):
    consec_count = 0
    #print(month_diff)
    for index , val in enumerate(month_diff):
        if index != 0 and val == 1:
                consec_count += 1
                if consec_count == 2:
                    return True
        else:
            consec_count = 0
​
    return False

Upvotes: 2

Related Questions