CODEWITHSUNDEEP
pythonpandas
user3180797
user3180797

Reputation: 31

Filtering Groups With Pandas

I am trying to add filters to groups using Pandas. In the baseball data below, I am looking to figure out the average time it takes from an initial 'N' to a final 'Y' in the inducted column. Essentially I am looking to calculate the length of each group that includes a 'Y' in the inducted column and has more than one row. Any tips would help! 

   playerID  yearid votedBy  ballots  needed  votes inducted category needed_note
2860  aaronha01    1982   BBWAA      415     312    406        Y   Player         NaN
3743  abbotji01    2005   BBWAA      516     387     13        N   Player         NaN
 146  adamsba01    1937   BBWAA      201     151      8        N   Player         NaN
 259  adamsba01    1938   BBWAA      262     197     11        N   Player         NaN
 384  adamsba01    1939   BBWAA      274     206     11        N   Player         NaN
 497  adamsba01    1942   BBWAA      233     175     11        N   Player         NaN
 574  adamsba01    1945   BBWAA      247     186      7        N   Player         NaN
2108  adamsbo03    1966   BBWAA      302     227      1        N   Player         NaN

Upvotes: 1

Views: 583

Answers (3)

slushy
slushy

Reputation: 3357

The filter method for the class DataFrameGroupBy operates on each subframe in the group. See help(pd.core.groupby.DataFrameGroupBy.filter). The data are:

print df
  inducted playerID
0        Y        a
1        N        a
2        N        a
3        Y        b
4        N        b
5        N        c
6        N        c
7        N        c

Example code:

import pandas as pd

g = df.groupby('playerID')
madeit = g.filter(
        lambda subframe:
                'Y' in set(subframe.inducted)).groupby('playerID')

# The filter removed player 'c' who never has inducted == 'Y'
print madeit.head()
           inducted playerID
playerID                    
a        0        Y        a
         1        N        a
         2        N        a
b        3        Y        b
         4        N        b

# The 'aggregate' function applies a function to each subframe
print madeit.aggregate(len)
          inducted
playerID          
a                3
b                2

Upvotes: 1

CT Zhu
CT Zhu

Reputation: 54340

I modified your dataset so that there are two such groups. One has 2 rows from N to Y, the other has 8 rows from N to Y. It depends on whether you count in the y containing rows. If not, it will have two groups, one contains 1 row and the other contains 7 rows. It also look like you don't have a time series column, so I guess that means the rows are evenly distributed time-wise. 

In [25]:

df=pd.read_clipboard()
print df
       playerID  yearid votedBy  ballots  needed  votes inducted category  needed_note 
3741  abbotji01    2005   BBWAA      516     387     13        N   Player          NaN 
2860  aaronha01    1982   BBWAA      415     312    406        Y   Player          NaN 
3743  abbotji01    2005   BBWAA      516     387     13        N   Player          NaN 
146   adamsba01    1937   BBWAA      201     151      8        N   Player          NaN 
259   adamsba01    1938   BBWAA      262     197     11        N   Player          NaN 
384   adamsba01    1939   BBWAA      274     206     11        N   Player          NaN 
497   adamsba01    1942   BBWAA      233     175     11        N   Player          NaN 
574   adamsba01    1945   BBWAA      247     186      7        N   Player          NaN 
2108  adamsbo03    1966   BBWAA      302     227      1        N   Player          NaN 
2861  aaronha01    1982   BBWAA      415     312    406        Y   Player          NaN 

In [26]:

df['isY']=(df.inducted=='Y')
df['isY']=np.hstack((0,df['isY'].cumsum().values[:-1])).T
In [27]:

print df.groupby('isY').count()
     playerID  yearid  votedBy  ballots  needed  votes  inducted  category  needed_note  isY 
0           2       2        2        2       2      2         2         2            0    2 
1           8       8        8        8       8      8         8         8            0    8 
[2 rows x 10 columns]

Say if you don't count the Y, the means can be calculated by :

df2=df.groupby('isY').count().isY-1
df2[df2!=1].mean()

Upvotes: 0

BushMinusZero
BushMinusZero

Reputation: 1292

I simulated my own data to make an easy test case of your problem. I create a set of players called df_inducted that includes players who were eventually inducted then by using the isin() function we can make sure to only consider them in the analysis. Then I find the min and max of their dates and average their differences.

import pandas as pd

df = pd.DataFrame({'player':['Nate','Will','Nate','Will'], 
                   'inducted': ['Y','Y','N','N'],
                   'date':[2014,2000,2011,1999]})

df_inducted = df[df.inducted=='Y']
df_subset = df[df.player.isin(df_inducted.player)]

maxs = df_subset.groupby('player')['date'].max()
mins = df_subset.groupby('player')['date'].min()

maxs = pd.DataFrame(maxs)
maxs.columns = ['max_date']
mins = pd.DataFrame(mins)
mins.columns = ['min_date']

min_and_max = maxs.join(mins)
final = min_and_max['max_date'] - min_and_max['min_date']

print "average time:", final.mean()

Upvotes: 0

Related Questions