CODEWITHSUNDEEP
pythonnumpymatplotlibscipy
DanHickstein
DanHickstein

Reputation: 6908

Numpy: averaging many datapoints at each time step

This question is probably answered somewhere, but I cannot find where, so I will ask here:

I have a set of data consisting of several samples per timestep. So, I basically have two arrays, "times", which looks something like: (0,0,0,1,1,1,1,1,2,2,3,4,4,4,4,...) and my data which is the value for each time. Each timestep has a random number of samples. I would like to get the average value of the data at each timestep in an efficient manner.

I have prepared the following sample code to show what my data looks like. Basically, I am wondering if there is a more efficient way to write the "average_values" function. 

import numpy as np
import matplotlib.pyplot as plt

def average_values(x,y):
    unique_x = np.unique(x)
    averaged_y = [np.mean(y[x==ux]) for ux in unique_x]
    return unique_x, averaged_y

#generate our data
times   = []
samples = []

#we have some timesteps:
for time in np.linspace(0,10,101):

    #and a random number of samples at each timestep:
    num_samples = np.random.random_integers(1,10)

    for i in range(0,num_samples):
        times.append(time)
        samples.append(np.sin(time)+np.random.random()*0.5)

times   = np.array(times)
samples = np.array(samples)

plt.plot(times,samples,'bo',ms=3,mec=None,alpha=0.5)
plt.plot(*average_values(times,samples),color='r')
plt.show()

Here is what it looks like: enter image description here

Upvotes: 3

Views: 1990

Answers (2)

elyase
elyase

Reputation: 40963

May I propose a pandas solution. It is highly recommended if you are going to be working with time series.

Create test data

import pandas as pd
import numpy as np

times = np.random.randint(0,10,size=50)
values = np.sin(times) + np.random.random_sample((len(times),))
s = pd.Series(values, index=times)
s.plot(linestyle='.', marker='o')

enter image description here

Calculate averages

avs = s.groupby(level=0).mean()
avs.plot()

enter image description here

Upvotes: 5

Jaime
Jaime

Reputation: 67417

A generic code to do this would do something as follows:

def average_values_bis(x, y):
    unq_x, idx = np.unique(x, return_inverse=True)
    count_x = np.bincount(idx)
    sum_y = np.bincount(idx, weights=y)

    return unq_x, sum_y / count_x

Adding the function above and following line for the plotting to your script

plt.plot(*average_values_bis(times, samples),color='g')

produces this output, with the red line hidden behind the green one:

enter image description here

But timing both approaches reveals the benefits of using bincount, a 30x speed-up:

%timeit average_values(times, samples)
100 loops, best of 3: 2.83 ms per loop

%timeit average_values_bis(times, samples)
10000 loops, best of 3: 85.9 us per loop

Upvotes: 9

Related Questions