What is the right way to manage namespaces in Python 3?

Question

Most of my programming background is in C++ and Java, but for professional reasons I'm starting to learn Python. One of the first things that I noticed was Python's new approach to packages and namespaces, but after googling around for awhile and doing some experimenting I thought I started to get the hang of it.

Then I read this, which indicated that everything I had just learned was wrong. So now I'm confused again, and I'm having trouble figuring out how I should organize my code.

---

Here's a very concrete question which I don't really know how to answer correctly for any version of Python, and in particular I don't know if the answer changed from Python 2 to 3. Let's say I want a class in a cloned github repo, stored in:

/Users/me/my_teams_code/some_project/package/IWantThisClass.py

But let's say I have some other projects here:

/Users/me/my_own_code/some_other_project1/...
/Users/me/my_own_code/some_other_project2/...
/Users/me/my_own_code/some_other_project3/...

each of which use IWantThisClass. One strategy seems to be to just manually add the correct directory to sys.path every time, but this is rather cumbersome - especially when I'm just experimenting in a Jupyter notebook. The solution that I settled upon after googling around is to add the package directory to my PYTHONPATH variable in bash; this works in the sense that "import IWantThisClass" works just fine. But I read in Coghlan's post above:

"This next trap exists in all current versions of Python, including 3.3, and can be summed up in the following general guideline: 'Never add a package directory, or any directory inside a package, directly to the Python path'."

So what is the "correct" way to solve this problem?

Answer 1

Basically, if package is the top-level directory of your package, you should add the directory containing package to sys.path. If what you want to import is not a package but a single module (i.e., just an individual .py file not in a package directory), then that module should be in a directory on sys.path/

Exactly how you should add these paths is another question. For situations where I want to add one-off things to the path, I find that the easiest thing to do is to create a .PTH file inside a "site directory" --- either the main site-packages directory or the user-specific site-packages directory. In that .PTH file you can list other directories which will then be added to site.path. So you can have your own little my_stuff.pth file that sets up sys.path with directories you need. This avoids having to add boilerplate code to each script to manually modify sys.path. You can read about how to do this in the documentation for the site module.

Another, perhaps better answer is that you should try to avoid adding specific directories to sys.path in order to access specific files. Rather, you should try to install the relevant packages, which will let Python set up the paths for you. If you're using in-development packages that you don't want to clutter your main site-packages will, you can use python setup.py develop or pip install --editable to install the packages "in-place". (That is, the package files will stay where they are, not be moved into the global Python site-packages, and the installer will update the appropriate .PTH files so that the newly-installed package is importable.)

In short, ultimately you need to move away from viewing it in terms of "this file wants this class from this other file". The unit that you should be concerned with is the installable package or module. If you're writing code that wants SomeClass, you should install the package/module that provides SomeClass and then import it; don't try to manipulate sys.path on an ad-hoc basis to target the individual file/class that you want without installing its package. (As mentioned, setup.py develop and pip install --editable allow you to do this while still keeping the actual files where you want.)