CODEWITHSUNDEEP
pythonlambda
user15964
user15964

Reputation: 2639

Why lambda function in list comprehension is slower than in map?

I made a test

%timeit list(map(lambda x:x,range(100000)))
%timeit [(lambda x:x)(i) for i in range(100000)]

gives

29 ms ± 4.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
51.2 ms ± 3.76 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Why in python lambda function is slower in list comprehension than in map?

Upvotes: 4

Views: 1246

Answers (2)

MisterMiyagi
MisterMiyagi

Reputation: 50076

TLDR: A comprehension evaluates its entire expression every time. A map evaluates its expression once, and applies the result every time.

The expression fed to map and comprehensions are treated differently. The important parts of what map and comprehensions are doing can be translated in the following way:

def map(func: 'callable', iterable):
    for item in iterable:
        yield func(item)

def comprehension(expr: 'code', iterable):
    for item in iterable:
        yield eval(expr)

The important difference is that map already receives a function, whereas comprehension receives an expression.

Now, lambda x:x is an expression that evaluates to a function.

>>> co = compile('lambda x: x', '<stackoverflow>', 'eval')
>>> co
<code object <module> at 0x105d1c810, file "<stackoverflow>", line 1>
>>> eval(co)
<function __main__.<lambda>(x)>

Notably, evaluating the expression to get the function is an action that takes time.

Since map and comprehensions expect different things, the lambda is passed to them differently. What the interpreter does can be expanded with explicit eval/compile:

>>> list(map(eval(compile('lambda x: x', '<stackoverflow>', 'eval')), range(10)))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(comprehension(compile('(lambda x: x)(item)', '<stackoverflow>', 'eval'), range(10))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The important difference should be clear now:

  • For map, the expression is evaluated once before it is passed into map. For each item, map calls the resulting function.
  • For comprehension, the expression is passed in unevaluated. For each item, comprehension constructs the function, then calls the resulting function.

It is important to note that map is not always faster than a comprehension. map benefits from creating/looking up a function once. However, function calls are expensive and a comprehension does not have to use a function.

In general, if the expression uses only names local to the comprehension, it is faster.

In [302]: %timeit list(map(lambda i:i,range(100000)))
     ...: %timeit [i for i in range(100000)]
8.86 ms ± 38.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
4.49 ms ± 35.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

If the expression itself has to dynamically do an expensive operation, map does not benefit.

In [302]: %timeit list(map(lambda i: (lambda x:x)(i),range(100000)))
     ...: %timeit [(lambda x:x)(i) for i in range(100000)]
19.7 ms ± 39.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
15.9 ms ± 43.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Upvotes: 4

Ma0
Ma0

Reputation: 15204

Because in your first code snippet the lambda is created only once and executed 100000 times while on the second it is created and executed in every iteration.

To be honest I am surprised that the difference is not even bigger but your two timings should grow further apart the largest the length of the iterable is.

As a sidenote, notice that even if you change the lambda to a built-in function that does not have to be created first but rather only looked-up, you still get the same trend:

> py -m timeit "list(map(float, range(10000)))"
200 loops, best of 5: 1.78 msec per loop

> py -m timeit "[float(x) for x in range(10000)]"
100 loops, best of 5: 2.35 msec per loop

An improvement can be achieved by binding the function to a variable, but the list(map()) scheme is still faster.

> py -m timeit "r=float;[r(x) for x in range(10000)]"
100 loops, best of 5: 1.93 msec per loop

As to why the binding to a local variable is faster, you can find more information here.

Upvotes: 7

Related Questions