Explain this list comprehension oddity in the general case and what are it's use cases?

Question

Yep. This happend. When I absent mindedly put an index on a in variable. Explain (it?) away. What is happening in the general case and what are it's use cases?

>>> [q for q[0] in [range(10),range(10,-1,-1)]]

Traceback (most recent call last):
  File "<pyshell#209>", line 1, in <module>
    [q for q[0] in [range(10),range(10,-1,-1)]]
NameError: name 'q' is not defined

>>> [q for q in [range(10),range(10,-1,-1)]]
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10,-1,-1),range(10)]]
[[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10,-1,-1),range(10)]]
[[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q in [range(10),range(10,-1,-1)]]
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]

Answer 1

The for loop compound statement reuses the target_list construct also used in assignments:

for_stmt ::=  "for" target_list "in" expression_list ":" suite

That's because for each iteration of the loop, the 'current' value is assigned to the target list. That also means you can assign to an item. Quoting the documentation:

Each item in turn is assigned to the target list using the standard rules for assignments,

In Python, this is perfectly normal:

q = [None]
q[0] = 'foobar'

and you can do the same in a for loop, provided the name q exists and supports item assignment. This is why your first attempt failed:

>>> [q for q[0] in [range(10),range(10,-1,-1)]]
Traceback (most recent call last):
  File "<pyshell#209>", line 1, in <module>
    [q for q[0] in [range(10),range(10,-1,-1)]]
NameError: name 'q' is not defined

There is no name q yet, so assigning to q[0] fails.

There probably is no real use case for using this in a for loop, but the advantage of reusing target_list here is that it makes the grammar and parser simpler, and lets you use assignment unpacking, e.g. assignment to multiple targets:

for key, value in mapping.items():
    # unpack two-value tuples returned by .items() into two names

Next, you executed a straight-up list comprehension:

>>> [q for q in [range(10),range(10,-1,-1)]]
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]

In Python 2, the local names of a list comprehension leak; they live in the scope the list comprehension is defined in. See Python list comprehension rebind names even after scope of comprehension. Is this right?

So the moment the above list comprehension is executed, the name q exists, still bound to the value of the last iteration:

>>> q
[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

This is a list object, so it supports item assignment. From there on out, what you do is item assignment to q[0] and re-referencing this one global q name:

>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> q
[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

You've put each of the two range()-produced lists in index 0 of q, but after the list comprehension is done, q[0] is bound to the last one being iterated over.

In Python 3, the item assignment in a for loop will still work, but list comprehensions no longer 'leak' local names as they are executed in a separate scope. Dict and set comprehensions, as well as generator expressions, are executed in a separate scope, always, regardless of Python versions.

Answer 2

There is no oddity in the way list comprehension is working. You need to know what the contents of your variables are, between the sequence of operations you are doing.

The first line in your snippet

[q for q[0] in [range(10),range(10,-1,-1)]]

errors out because the interpreter does not know what q is yet. And you are trying to access the first element of an unknown variable by doing q[0]. You need to be doing:

[q[0] for q in [range(10),range(10,-1,-1)]]

As to why it works in the next sequence of operations?

After you do:

[q for q in [range(10),range(10,-1,-1)]]

q is now a variable with value [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0] and hence your next operation:

[q for q[0] in [range(10),range(10,-1,-1)]]

does not fail. If in the above operation [q for q[0] in [range(10),range(10,-1,-1)]] you use another variable p instead of q, you would get the original error you saw the first time.