CODEWITHSUNDEEP
pythonpandasdataframegroup-bypandas-groupby
VivianT
VivianT

Reputation: 43

Find minimum daily value using pandas GroupBy or pivot_table

I have a Dataframe obtained from a csv file (after some filtering) that looks like this:

 df3.head(n = 10)

        DateTime            Det_ID  Speed
16956   2014-01-01 07:00:00 1201085 65.0
16962   2014-01-01 07:00:00 1201110 69.5
19377   2014-01-01 08:00:00 1201085 65.0
19383   2014-01-01 08:00:00 1201110 65.0
21798   2014-01-01 09:00:00 1201085 65.0
21804   2014-01-01 09:00:00 1201110 65.4
75060   2014-01-02 07:00:00 1201085 64.9
75066   2014-01-02 07:00:00 1201110 66.1
77481   2014-01-02 08:00:00 1201085 65.0
77487   2014-01-02 08:00:00 1201110 62.5

This represents the speeds measured by different detectors (two for now) at various times of day. I have converted the DateTime column to a datetime object.

I need to know for each detector, the minimum daily value of the speed.

Basically, something like this, which I can then use to build a heat map.

df4 = df3.pivot_table(index='DateTime',columns='Det_ID',aggfunc=min)
df4.head()

                      Speed
Det_ID             1201085  1201110
DateTime        
2014-01-01 07:00:00 65.0    69.5
2014-01-01 08:00:00 65.0    65.0
2014-01-01 09:00:00 65.0    65.4
2014-01-02 07:00:00 64.9    66.1
2014-01-02 08:00:00 65.0    62.5

Clearly, the way I've used the pivot table is incorrect as I'm getting multiple values of daily speeds, not just one. I suspect it is because the minimum is being calculated over each unique DateTime field, not just the for the date part.

Also trying groupby options.

list(df3.groupby(['DateTime'], sort = False)['Speed'].min())

But it just gives a list of numbers, without any other columns.

65.0,
 65.0,
 65.0,
 64.900000000000006,
 62.5,
 64.200000000000003,
 54.700000000000003,
 62.600000000000001,
 64.799999999999997,
 59.5,

etc.

How do I isolate just the date part in the DateTime field? Am I even going in the right direction? Thanks.

Upvotes: 2

Views: 2141

Answers (2)

BENY
BENY

Reputation: 323226

Or using unstack

df.DateTime = df.DateTime.dt.strftime('%m/%d/%Y')
df.groupby(['DateTime','Det_ID']).Speed.min().unstack()
Out[300]: 
Det_ID      1201085  1201110
DateTime                    
01/01/2014     65.0     65.0
01/02/2014     64.9     62.5

Upvotes: 1

cs95
cs95

Reputation: 402483

Call .dt.strftime and reformat your DateTime column.

df.DateTime = df.DateTime.dt.strftime('%m/%d/%Y')
df

        DateTime   Det_ID  Speed
16956  01/01/2014  1201085   65.0
16962  01/01/2014  1201110   69.5
19377  01/01/2014  1201085   65.0
19383  01/01/2014  1201110   65.0
21798  01/01/2014  1201085   65.0
21804  01/01/2014  1201110   65.4
75060  01/02/2014  1201085   64.9
75066  01/02/2014  1201110   66.1
77481  01/02/2014  1201085   65.0
77487  01/02/2014  1201110   62.5

Now, call pivot_table:

df = df.pivot_table(index='DateTime', columns='Det_ID', values='Speed', aggfunc=np.min)
df
Det_ID      1201085  1201110
DateTime                    
01/01/2014     65.0     65.0
01/02/2014     64.9     62.5

Upvotes: 2

Related Questions