CODEWITHSUNDEEP
pythonpandas
user2915556
user2915556

Reputation: 199

Averaging along DataFrame columns with some missing rows

I have a DataFrame object with several columns like that:

+--------+---------------------+-------+-------+-------+
|        |        Date         | temp1 | temp2 | temp3 |
+--------+---------------------+-------+-------+-------+
|  17687 | 2013-10-21 00:04:47 | 14.50 | 13.55 | 43.06 |
|  48117 | 2013-10-21 00:18:07 | 14.64 | 13.42 | 37.48 |
|  76509 | 2013-10-21 00:33:51 | 14.32 | 13.55 | 27.26 |
| 102769 | 2013-10-21 00:48:24 | 14.38 | 13.30 | 37.10 |
| 133862 | 2013-10-21 01:04:36 | 14.77 | 13.11 | 28.25 |
| 162882 | 2013-10-21 01:18:14 | 14.50 | 13.98 | 39.71 |
| 191902 | 2013-10-21 01:34:11 | 14.39 | 13.08 | 28.69 |
| 220922 | 2013-10-21 01:48:38 | 14.31 | 13.00 | 43.56 |
| 249942 | 2013-10-21 02:04:26 | 14.10 | 13.94 | 39.79 |
| 278962 | 2013-10-21 02:18:13 | 14.01 | 13.55 | 23.46 |
| 307982 | 2013-10-21 02:34:04 | 14.00 | 13.21 | 44.94 |
| 337002 | 2013-10-21 02:48:27 | 14.81 | 13.38 | 29.44 |
+--------+---------------------+-------+-------+-------+
...
+--------+---------------------+-------+-------+-------+
|  15531 | 2013-10-22 00:05:20 | 14.84 | 13.07 | 30.25 |
|  44149 | 2013-10-22 00:18:11 | 14.35 | 13.22 | 51.02 |
| 102685 | 2013-10-22 00:49:34 | 14.46 | 13.98 | 24.17 |
| 127960 | 2013-10-22 01:04:02 | 14.07 | 13.49 | 30.74 |
| 186892 | 2013-10-22 01:34:14 | 14.75 | 13.01 | 45.77 |
| 214754 | 2013-10-22 01:48:17 | 14.35 | 13.03 | 40.75 |
| 240236 | 2013-10-22 02:02:39 | 14.31 | 13.28 | 34.88 |
| 507942 | 2013-10-21 02:34:04 | 14.87 | 13.62 | 50.16 |
| 111987 | 2013-10-21 02:48:27 | 14.74 | 13.63 | 51.36 |
+--------+---------------------+-------+-------+-------+

The problem is to find average values of temp1, temp2 and temp3 for a period of time (say, 2 days) over the same intervals (for that example - 15 minutes). There are two issues: (1) some rows were missed; (2) temperatures were measured in slightly different (+- 3 minutes in whole dataset and 2 minutes in that particular example) time.

As of now, my solution is based on 2 steps. First, find maximum number of intervals along the day (look through all days in base). Create a new DataFrame object with corresponding number of rows. Second, go through dataset and add values to correspond row if current date within 3 minutes. Unfortunately, it's a bit of slow because of iterative nature. I'm trying to find a way to do it faster.

Any suggestions?

Thanks

P.S. It would be nice to see a result like that (first column is some averaged time):

+---------+-------+-------+-------+
|  Time   | temp1 | temp2 | temp3 |
+---------+-------+-------+-------+
| 0:05:00 | 14.67 | 13.31 | 36.66 |
| 0:18:00 | 14.50 | 13.32 | 44.25 |
| 0:34:00 | 14.32 | 13.55 | 27.26 |
| 0:49:00 | 14.42 | 13.64 | 30.64 |
| 1:04:00 | 14.42 | 13.30 | 29.50 |
| 1:18:00 | 14.50 | 13.98 | 39.71 |
| 1:34:00 | 14.57 | 13.05 | 37.23 |
| 1:48:00 | 14.33 | 13.02 | 42.16 |
| 2:03:00 | 14.21 | 13.61 | 37.34 |
| 2:18:00 | 14.01 | 13.55 | 23.46 |
| 2:34:00 | 14.44 | 13.42 | 47.55 |
| 2:48:00 | 14.78 | 13.51 | 40.40 |
+---------+-------+-------+-------+

Upvotes: 2

Views: 83

Answers (1)

alko
alko

Reputation: 48317

Does following solve your task?

import datetime
from collections import defaultdict

def avg(lst):
    return sum(lst)/len(lst)

Define some sample data

def s2dt(s):
    fmt = '%Y-%m-%d %H:%M:%S'
    return datetime.datetime.strptime(s, fmt)

data = [(s2dt('2013-10-21 00:04:47'), 14.50, 13.55, 43.06),
        (s2dt('2013-10-21 00:18:07'), 14.64, 13.42, 37.48),
        (s2dt('2013-10-22 00:05:20'), 14.84, 13.07, 30.25),
        (s2dt('2013-10-22 00:18:11'), 14.35, 13.22, 51.02)]

Define a function that will group times by periods

def coarse(dt, granularity):
    residue = dt.minute % granularity
    if residue:
        residue = granularity-residue
    dt = dt + datetime.timedelta(minutes=residue, seconds=-dt.second,
                    microseconds=-dt.microsecond)
    t = dt.time()
    return t

Group data by intervals

groupings = defaultdict(list)
for dt, t1, t2, t3 in data:
    groupings[coarse(dt, 15)].append([t1, t2, t3])

Calculate average

averages = dict((k, map(avg, zip(*v))) for k, v in groupings.iteritems())

and get 

>>> for ct, values in sorted(averages.iteritems()):
...   print ct, ', '.join(map(lambda x: '%.2f' % x, values))
00:15:00 14.67, 13.31, 36.66
00:30:00 14.50, 13.32, 44.25

Upvotes: 1

Related Questions