CODEWITHSUNDEEP
pythonmultithreadingperformancetime-complexitylist-comprehension
gm1
gm1

Reputation: 265

Boosting time for multiple for loops in list comprehensions

I am looking for a method to lower the execution time that Python 3.5 takes for two for-loops in a list comprehension that looks something like this:

[[(k1-k2)**power for k2 in range(m,n)] for k1 in range(m,n)]

Upvotes: 2

Views: 167

Answers (1)

Nelewout
Nelewout

Reputation: 6574

So, I started off with your current approach and found that while it does work, it is quite slow. My first attempt involved simply transforming your list comprehension to a suitable approach with numpy arrays. That was about three times faster than your original approach, but that's when I noticed something quite beautiful: this is a symmetric Toeplitz matrix. From that wiki page:

In linear algebra, a Toeplitz matrix or diagonal-constant matrix, named after Otto Toeplitz, is a matrix in which each descending diagonal from left to right is constant.

I first used the default scipy implementation of a Toeplitz matrix, but that approach was unnecessary slow for your problem. So I wrote myself a similar approach, which is the third attempt below.

Methodology

I ran 10 tests for each approach, with each individual test comprising 1000 runs. I set the parameters to m = 10, n = 100. The results can be found in the table below:

    Your approach       Numpy #1     Numpy #2     Numpy #3
1        4.573965       1.432406     1.060242     0.186767
2        4.341466       1.432237     1.060404     0.186872
3        4.442438       1.434460     1.144850     0.183120
4        4.318919       1.456928     1.072072     0.185626
5        4.249392       1.450684     1.072217     0.183273
6        4.202730       1.508863     1.070299     0.183019
7        4.224226       1.457543     1.065354     0.183591
8        4.234505       1.432971     1.082438     0.185711
9        4.256538       1.431828     1.080051     0.184290
10       4.241055       1.557204     1.083070     0.185845

AVG      4.308523       1.459512     1.079100     0.184811
STD      0.117433       0.041693     0.024538     0.001521

The scipy Toeplitz approach (Numpy #2 in the table) is almost four times faster than your current approach, but all these results are dwarfed by the third and final approach: a whopping 23 times speed up over your initial implementation!

Now, since you were interested in the time complexity, I let n vary, keeping m = 10. The results for each approach can be found in the figure below:

Time complexity of the different approaches.

Clearly, the third approach is the way to go!

Code

Full code:

import timeit
import numpy as np
from scipy.linalg import toeplitz


def your_approach(m, n):
    print("\n\tlist comprehension")
    k = range(m, n)
    for i in range(1, 11):
        start = timeit.default_timer()
        for j in range(1, 1001):
            data_list_comp = [[(k1 - k2) ** 2 for k2 in k] for k1 in k]
        print("\t{}".format(timeit.default_timer() - start))
    return data_list_comp


def numpy1(m, n):
    print("\n\tnumpy")
    k_n = np.array(range(m, n))
    for i in range(1, 11):
        start = timeit.default_timer()
        for j in range(1, 1001):
            data_numpy = [list((k_n - x) ** 2) for x in k_n]
        print("\t{}".format(timeit.default_timer() - start))
    return data_numpy


def numpy2(m, n):
    print("\n\ttoeplitz")
    k_n = np.array(range(0, n - m)) ** 2
    toep = toeplitz(k_n)
    for i in range(1, 11):
        start = timeit.default_timer()
        for j in range(1, 1001):
            data_numpy = [list(toep[:, i]) for i in range(n - m)]
        print("\t{}".format(timeit.default_timer() - start))
    return data_numpy

def numpy3(m, n):
    print("\n\ttoeplitz2")
    k_n = list(np.array(range(0, n - m)) ** 2)  # can obviously be done without numpy, but I was a bit lazy. :)
    for i in range(1, 11):
        start = timeit.default_timer()
        for j in range(1, 1001):
            data_numpy = [(k_n[i::-1] + k_n[1:-i]) if i != 0 else k_n for i in range(0, n - m)]
        print("\t{}".format(timeit.default_timer() - start))
    return data_numpy

m = 10

for n in [25, 50, 100, 150, 200]:
    assert your_approach(m, n) == numpy1(m, n) == numpy2(m, n) == numpy3(m, n)

Upvotes: 6

Related Questions