Avoiding `sys.path.append(..)` for imports

Question

This isn't the first time I am cringing over imports in Python. But I guess this one is an interesting use case, so I thought to ask it here to get a much better insight. The structure of my project is as follows:

sample_project
   - src
        - __init__.py
        - module1
           - __init__.py
           -  utils.py
        - module2
           - __init__.py 
           - models.py
        - app.py

The module1 imports methods from module2 and app imports method from all the other. Also, when you run the app it needs to create a folder called logs outside of src folder. There are now to ways to run the app:

1. From inside src folder flask run app
2. From outside of src folder flask run src.app

To make sure that I don't get import errors because of the change of the top level module where the app is started, I do this:

import sys
sys.path.append("..")

Is there any better solution to this problem?

Answer 1

Pythonic Solution: Use packages and modules as they are intended to be used

Ask yourself, why did you want to create a src directory?

I would suggest that more than likely you wanted to follow a convention you knew from another language. (Maybe Java, maybe C, C++, or something else.)

However, if you use Python packages in the way they are intended to be used, there is a far simpler solution.

First lets review a few key points.

• To run the Python file main.py, you run it using the Python interpreter like so: python3 main.py.
• When the Python interpreter starts, it adds the current working directory to its path. (via sys.path)
• It will also add the directory containing the module to be run to its path
• More information can be found in this documentation page: https://docs.python.org/3/tutorial/modules.html
• One issue with that documentation page is it explains how to modify sys.path from within Python code. The issue with that is it gives developers the idea that this is possible and therefore should be used as a solution to import and path problems when it should not.
• The Python interpreter will search the sys.path and PYTHONPATH directories list for modules and packages to resolve when it sees an import statement
• A Python package is a directory with an __init__.py file
• A Python module is just a regular Python file
• The __init__.py file is there to signal to the Python interpreter that it needs to recursively search subdirectories for more Python packages and modules. This is why an __init__.py is usually empty.
• Without an __init__.py the Python interpreter will simply ignore a directory
• This rule is actually an optmization to prevent the interpreter from becoming slow to start up if there are a large number of subdirectories and files to search
• From this we conclude that all local source code should be resolvable from the same directory as the one used to run the target module (main.py)

With that information, you can re-structure your project:

sample_project/
    my_python_package/
        __init__.py
        sub_package_1/
            __init__.py
            utils.py
        sub_package_2/
            __init__.py 
            models.py
    app.py

Run app.py from the directory sample_project: python3 app.py

You can actually go further. If your project becomes very large, it sometimes makes sense to run modules within packages using python3 -m some_package.some_module. Then everything, including app.py becomes a package. I don't think you need this in this particular case, but if you have large numbers of "executable" Python files which are better grouped into a set of directories, then this is the approach to take.

Note that:

• This solution is simple (bordering on trivial, if not necessarily that obvious)
• There is no src directory. Forget about src. This works well in other languages, it doesn't fit into the Python model for how a project should be structured
• You did not need to modify PYTHONPATH
• You did not need to modify sys.path
• You did not need to write extra code to be able to resolve imports
• This solution is easy to understand, it is straight forward and has minimal complexity

---

An experiment to learn about PYTHONPATH and sys.path

You can find out what PYTHONPATH and sys.path are set to with a short experimental code:

$ cd ~
$ mkdir python-path-test
$ touch python-path-test/main.py 

# main.py

import os
import sys

print(f'PYTHONPATH:')
for string in os.environ.get('PYTHONPATH').split(';'):
    print(string)

print(f'sys.path:')
for string in sys.path:
    print(string)

$ export PYTHONPATH=`pwd`
$ python3 python-path-test/main.py
PYTHONPATH:
/home/username
sys.path:
/home/username
/home/username/python-path-test
/usr/lib/python311.zip
/usr/lib/python3.11
/usr/lib/python3.11/lib-dynload
/usr/local/lib/python3.11/dist-packages
/usr/lib/python3/dist-packages
/usr/lib/python3.11/dist-packages

---

Further explanation in regards to other answers

Let me address the issues with the other answers here. All of the answers provided will work, but none of them take the simplest and "most obviously correct" approach.

The reason for this is the "most obviously correct" approach is not that obvious, especially if you come to Python from other languages where things work differently.

Just to say as well - it took me a long time to figure out the solution to the exact same problem which is shown in the question and I only figured out the solution when I went to work for a firm where someone else had figured this out before me.

Also: None of this is really explained on any documentation page anywhere, so it is hardly surprising that most people get it wrong, or do something unneccessarily complex when it isn't needed.

Overview of other solutions:

So far several other solutions have been proposed:

1. Use setuptools and virtual environments to manage what is known as an "editable install".

I don't like this for two reasons: It is more work than is necessary, and you are pretending that some local source code is a PIP package, when it isn't. It just seems like a bizzare thing to do. (This is exactly what I used to do before realizing there is an easier way.)

2. Write Python code to modify the sys.path or PYTHONPATH environment variable

I don't like this because it is a hack:

• The PYTHONPATH environment variable is intended to be used to store the locations of installed packages on your system
• It should be a semi-permenant thing which doesn't change (often)
• This is similarly the case with sys.path
• The other reason modifying PYTHONPATH is bad is because you are embedding (hiding) some code within your project which does unexpected things
• PYTHONPATH should be managed by the Operating System, or at least by the user in a shell
• In my experience, twiddling things which should be managed by your operating system from within code frequently leads to hard to find bugs and hard to understand code

3. Modify PYTHONPATH from a shell

This is better than the above proposal of modifying it from with Python code, but it just isn't necessary, for the reasons I explained above.

---

Appendix:

To give a little further helpful information. Some languages (and corresponding build tools) are designed with maximum flexibility. Others contain built-in rules which constrain how files and folders should be arranged for the build system to work property. These rules are not always explicit or obvious.

cmake is a good example of a build system which offers maximum flexibility. Many projects contain a src directory, under which all the C/C++ code lives. The reason for this is cmake facilitates using explicit and arbitrary paths to configure the build.

On the other hand, the Rust module system is much more constrained. The existence of a directory "creates" a module (or submodule). Cargo and Rust require you to use the filesystem in a constrained way to get the modular structure you want.

Julia is more similar to C++ in that modules are explicit - there is a module keyword, and this is the only way to create a module. It also has include which can take an arbitrary path - although using the Julia build system in an arbitrary way is not recommended, just as it would not be recommended with cmake.

Finally, Python is a bit more tricky. Similarly to Julia, the build system needs to be told how and where modules can be loaded from. It is generally better not to add lots of arbitrary hard-coded paths to the code or build system. Rather, avoiding this and working with what the language offers natively is preferable.

In both the case of Julia and Python, this means that the interpreter/runtime should be able to load your code without adding additional paths.

With this constraint, you will write a much simpler project structure.

To answer some questions in the comments

Here is what one of my Python projects looks like.

python_project_root_directory/
  .vscode/
    settings.json
  .venv/
  lib_something/
    __init__.py
    lib_something_files.py
  the_main_module
    __main__.py
    __init__.py # might not be requried
    main.py # called from __main__.py
  tests/
    some_group/
      test_something.py
      test_another_thing.py
    another_group/
      test_more_things.py
  Dockerfile

Dockerfile

Note: Does not use .venv, because a Docker container is its own isolated environment. You can use a .venv if you want. Change the command to cmd ["./.venv/bin/python3", "-m", "the_main_module"].

from python:3.12-bookworm
... other stuff ...
run pip3 install --no-cache-dir --upgrade -r requirements.txt
cmd ["python3", "-m", "the_main_module"]

If rather than wanting to run a main module, you want to run a python file as "main", change to cmd ["python3", "main.py"].

settings.json

{
    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
}

Answer 2

I wrote a package to solve this nasty issue, Sysappend:

https://pypi.org/project/sysappend/

This appends every folder in your repository to the sys.path variable, so Python is able to import any folder.

If you use this package, you don't need to pollute your project with __init__.py files, and you will unlock relative imports (both parent and child folders) from anywhere, without all the fuss. You will get correctly working debug, test and deploy features without going mad. At the same time, you get correct type-hinting, syntax highlighting, and module highlights.

This package does require you to add a short one-liner to the top of all your Python files (it doesn't need to be all files -- but it's convenient and can be enforced with CI/CD, GitHub actions or simply added via a quick copy-paste).

In my opinion, this solves the super-nasty import and relative package management in an developer-friendly way which basic Python fails to deliver.

How to use it

1. pip install sysappend
2. Add an if True: import sysappend; sysappend.all() statement at the top of every python file in your repo. (It doesn't need to be every file, but it's just easier for convenience, and the function caches results avoiding redundant computation, so it doesn't slow the code down).

Recommended folder referencing when importing

You should always try to reference the folders in the same way.
For example, use one (or few) agreed-upon primary source code folder from which to reference the sub-directories and stick to that convention.

E.g., if your directory looks like the below, you could pick src to be the primary source code folder:

sample_project/
    src/
        sub_folder_1/
            utils.py
        sub_folder_2/
            models.py
    app.py

So, when you write imports, they should start from the primary src folder.

E.g., in your app.py file, you should do:

from src.sub_folder_1.utils import somefunction

and you should do the same thing in models.py:

from src.sub_folder_1.utils import somefunction

Do not use a sub_folder as a starting name for an import, i.e., do not do from sub_folder_1.utils import somefunction. Although it will may still work in most cases, it may fail when you do type comparisons or deserialization, as Python looks at the import path to compare/deserialize types.

Quality of life setting for editors like VSCode

If you're using an editor like VSCode, you may want to add the main primary source code folder(s) to your settings.json file, like:

    "python.autoComplete.extraPaths": [
        "./src",
    ]

This will help the autocomplete to reference the folders always starting from src, in the same way as explained above -- i.e., you won't get automatic completion that attempt to do imports from sub folders.

Answer 3

After doing research (here1, here2, here3, here4, here5, here6), I come up with the better solution at this time. This is in each python file, you can add its current path before import. The example code below:

import os
import sys
if os.path.dirname(os.path.abspath(__file__)) not in sys.path:
    sys.path.append(os.path.dirname(os.path.abspath(__file__)))

Answer 4

The pythonic solution for the import problem is to not add sys.path (or indirectly PYTHONPATH) hacks to any file that could potentially serve as top-level script (incl. unit tests), since this is what makes your code base difficult to change and maintain. Assume you have to reorganize your project structure or rename folders.

Instead this is what editable installs are made for. They can be achieved in 2 ways:

1. pip install --editable <path> (requires a basic setup.py)
2. conda develop <path> (requires the conda-build package)

Either way will add a symlink into your site-packages folder and make your local project behave as if it was fully installed while at the same time you can continue editing.

Always remember: KEEP THINGS EASY TO CHANGE

Answer 5

Wraps the flask command into a small script in the sample_project directory and set PYTHONPATH according to your project:

#!/bin/env bash

# Assuming script is sample_project
path=`dirname ${BASH_SOURCE[0]}`
full_path=`realpath "$p"`
export PYTHONPATH=$full_path/src:$PYTHONPATH

flask run app

You can also switch current directory to a working directory.

But it is best to package your project using setuptools and install it (possibly in developpement mode), in user space according to PYTHONUSERBASE or in virtual environment.

Answer 6

Take a look at the Python import system documentation and at the PYTHONPATH environment variable.

When your code does import X.Y, what the Python runtime does is look in each folder listed in your PYTHONPATH for a package X (a package simply being a folder containing an __init__.py file) containing a Y package.

Most of the time, appending to sys.path is a poor solution. It is better to take care of what your PYTHONPATH is set to : check that it contains your root directory (which contains your top-level packages) and nothing else (except site-packages which i). Then, from wherever you run your commands, it will work the same (at least for imports, os.cwd is another problem).

Depending of the way to run your Python scripts, . may be the only relevant paths in it, so that it depends on your current directory, requiring to append .. if you run it from inside one of your top-level package.

And maybe you should not run your scripts from a directory that is not the root of your project ?

TL;DR : a good PYTHONPATH makes for way less import errors.