Unpacking os.walk inconsistent errors for different filepath

Question

Basic Python question.

Am playing with os.walk function and see inconsistent error messages. Sample o/p below:

Python 3.6.2 (default, Jul 18 2017, 14:08:57)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> root, dir, files = os.walk('.')  # ------> **no error**
>>> root, dir, files = os.walk('/')  # ------> **error #1**
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: too many values to unpack (expected 3)
>>> root, dir, files = os.walk('/usr/bin') # ------>**error #2**
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: not enough values to unpack (expected 3, got 1)
>>>

My understanding is os.walk returns generator, so error #2 makes sense for me. One generator value is returned, but I'm unpacking to 3 values.

So:

1. Why I don't get any error when I os.walk('.')
2. Why error #2 and #1 are different.

EDIT 1:

Played around a bit more and found that the location where python is run affects the execution, since the meaning of "." varies.

raghu$ pwd
/Users/raghu/PycharmProjects/Fib
raghu$ python
Python 3.6.2 (default, Jul 18 2017, 14:08:57)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> root, dir, files = os.walk('.')  # ------>**no error**
>>> ^D
raghu$ cd ..
raghu$ pwd
/Users/raghu/PycharmProjects
raghu$ python
Python 3.6.2 (default, Jul 18 2017, 14:08:57)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> root, dir, files = os.walk('.')  # ------>**error**
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: too many values to unpack (expected 3)

Answer 1

When you unpack, you exhaust the iterable on the right side. You aren't getting a dirpath, dirname, filenames tuple back which you assign to root, dirs, filenames.

An attempt to assign the first result (the first yield, that is) of os.walk(<value) to root is done, then, the second result (second yield) to dirs and then, finally, the third yield to file. If more results (or less!) are yielded, you get your errors. If exactly three are returned, it seemingly works.

So, in the first case it was simply just luck and 3 results were yielded back to you which got assigned (3 tuples containing the dirpath, dirname, filenames!).

'/usr/bin' usually doesn't contain any subdirectories, that's why you get the different error there (only 1 three tuple yield is performed).

With '/' you also get a greater number of three element tuples returned, so you get the appropriate error.

You can take a glampse of this by examining the number of items the generator yields:

>>> from os import walk
>>> len(list(walk('.')))
2300
>>> len(list(walk('/')))  # this is really dumb
127242
>>> len(list(walk('/usr/bin')))
1

As you can see in the last example, only one 3-element tuple is yielded. If you find a folder for which the len(list(walk('folder'))) == 3, the unpacking will work.

Answer 2

Each value produced by (read: while iterating over) os.walk(...) is a 3-tuple. Your code acts like the function returns a single such 3-tuple, but what it really means is "expect exactly 3 directories in this directory tree and give (misleading) names to the (tuple of) information about each".

Answer 3

From the docs:

For each directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath, dirnames, filenames).

So in each of these cases, there may be a different amount of tuples that is yielding which is causing your problems.