CODEWITHSUNDEEP
pythonnumpy
HenrikW
HenrikW

Reputation: 57

Speeding up loop in python

I running into some performance problems and I would appreciate a quick look at this.

I'm reading frequency domain data and I need to make a data cube for compensating the start frequencies in time domain.

The way I do this now is:

compdata = [[np.exp(t_range*sflist[y]) for y in x] for x in sfcube]

where 

t_range = 1j*2*np.pi*time_axis_ns
time_axis_ns = np.array([x*delta_time_ns for x in xrange(number_of_points_for_fft)])
sflist = array([ 29500000.,  30500000.])

sfcube is a NxM array with indexes for sflist.

Calculating compdata is by far the slowest part of my program now, any ideas for optimization?

Upvotes: 2

Views: 204

Answers (5)

Jaime
Jaime

Reputation: 67427

Here's your calculation and an alternative, more numpythonic take on it:

compdata = [[np.exp(t_range*sflist[y]) for y in x] for x in sfcube]
compdata2 = np.take(np.exp(sflist[:, None] * t_range), sfcube, axis=0)

With this sample data set based on your question:

sfcube = np.random.randint(2, size=(100, 100))
delta_time_ns = 100
number_of_points_for_fft = 1024
time_axis_ns = np.array([x*delta_time_ns for x in xrange(number_of_points_for_fft)])
t_range = 1j*2*np.pi*time_axis_ns
sflist = np.array([ 29500000.,  30500000.])

I get these timings:

In [3]: %timeit compdata = [[np.exp(t_range*sflist[y]) for y in x] for x in sfcube]
1 loops, best of 3: 1.76 s per loop

In [4]: %timeit compdata2 = np.take(np.exp(sflist[:, None] * t_range), sfcube, axis=0)
10 loops, best of 3: 72.2 ms per loop

That's a 20x speed-up, although your result will change, depending on the actual sizes of sflist and sfcube. And what you get is a 3D numpy array, not a list of lists of arrays, although they are interchangeable in most settings, and of course:

In [5]: np.allclose(compdata, compdata2)
Out[5]: True

Upvotes: 1

Jdog
Jdog

Reputation: 10721

Firstly, you can create your time_axis_ns array using:

time_axis_ns = np.arange(number_of_points_for_fft)*delta_time_ns

Then rather than you iterating over indices in sfcube:

sflist[sfcube]

Should give the same result. Then do your np.exp(t_range* on the result of this.

Upvotes: 0

vartec
vartec

Reputation: 134601

Quick win would be to parallelize it across all your CPU cores (+HT if available) using multiprocessing module.

from multiprocessing import Pool
pool = Pool(16)  ## use total number of cores, x2 if HT is supported.

def f(y):
    return np.exp(t_range*sflist[y])

compdata = [pool.map(f, x) for x in sfcube]

Of course this will get you 4-fold to 16-fold speed improvement (depending on number of cores). Still, if that's not good enough, you need to figure out how to do that with vector/matrix operations.

Upvotes: 1

mbatchkarov
mbatchkarov

Reputation: 16049

If you have an array of indices, you can use it to vectorize the calculation:

In [1]: a = numpy.random.random((4,4))

In [2]: a
Out[2]: 
array([[ 0.08746418,  0.57716752,  0.85863208,  0.42931856],
       [ 0.90259153,  0.19590983,  0.73075288,  0.88976529],
       [ 0.71843135,  0.0169308 ,  0.98336314,  0.22227009],
       [ 0.93776916,  0.58943769,  0.55040806,  0.92192396]])

In [3]: a[0]
Out[3]: array([ 0.08746418,  0.57716752,  0.85863208,  0.42931856])

In [7]: index=[(0,1), (0,1)]

In [8]: a[index]
Out[8]: array([ 0.08746418,  0.19590983])

In [9]: a[index] = 100*a[index]

In [10]: a[index]
Out[10]: array([  8.74641837,  19.59098276])

This is usually much faster than using Python lists. Note that a is a NumPy array from the start, and the the indices I use are in the form a[list_of_rows, list_of_columns]

Upvotes: 0

rocker_raj
rocker_raj

Reputation: 157

You can use Python-psyco library

http://psyco.sourceforge.net/
http://en.wikipedia.org/wiki/Psyco

or any other similar library

It may speed up your program.

Upvotes: 0

Related Questions