How do I properly import python modules in a multi directory project?

Question

I have a python project with a basic setup that looks like this:

imptest.py

utils/something.py
utils/other.py

Here's what's in the scripts:

imptest.py

#!./venv/bin/python

import utils.something as something
import utils.other as other

def main():
    """
    Main function.
    """

    something.do_something()
    other.do_other()

if __name__ == "__main__":
    main()

something.py

#import other

def do_something():
    print("I am doing something")


def main():
    """
    Main function
    """

    do_something()
    #other.do_other()

if __name__ == "__main__":
    main()

other.py

def do_other():
    print("do other thing!")

def main():
    """
    Main function
    """

    do_other()

if __name__ == "__main__":
    main()

imptest.py is the main file that runs and calls the utils functions occasionally for some things.

And as you can see, I have commented out some lines in "something.py" where I am importing "other" module for testing.

But when I want to test certain functions in something.py, I have to run the file something.py and uncomment the import line.

This feels like a bit of a clunky way of doing this.

If I leave the

import other

uncommented and run imptest.py, I get this error:

Traceback (most recent call last):
  File "imptest.py", line 5, in <module>
    import utils.something as something
  File "...../projects/imptest/utils/something.py", line 3, in <module>
    import other
ModuleNotFoundError: No module named 'other'

What's a better way of doing this?

Answer 1

I think what you're looking for is this. In order to be able to import a library and use it as executable you can do this. First, you should make utils a package. This means creation of __init__.py inside of it.

It's content should be this:

→ cat utils/__init__.py
import os
import sys
sys.path.append(os.path.dirname(os.path.realpath(__file__)))

This will make the import statement work as from utils import something, other in imptest.py and also import other in something.py.

The final piece is as follows:

→ tree
.
 |-imptest.py
 |-utils
 | |-__init__.py
 | |-other.py
 | |-something.py


→ cat imptest.py
#!./venv/bin/python
from utils import something, other

def main():
    """
    Main function.
    """

    something.do_something()
    other.do_other()

if __name__ == "__main__":
    main()

→ cat utils/something.py
import other

def do_something():
    print("I am doing something")


def main():
    """
    Main function
    """

    do_something()
    other.do_other()

if __name__ == "__main__":
    main()

→ python3 imptest.py
I am doing something
do other thing!

→ python3 utils/something.py
I am doing something
do other thing!

This trick makes both import statements valid. It basically adds the utils directory path to the search path, regardless where it is located. As such, import other will work. The __init__.py makes the utils directory a package;

Answer 2

PavanSkipo has solved the ImportError using relative path here but the files can then only be invoked as modules through python -m utils.something - whereas python utils/something.py will throw an error. The workaround using sys.path as PavanSkipo has mentioned in not ideal in my opinion ¹.

OP's use of the statement if __name__=='__main__' suggest that the files are meant to be run directly in some ways - perhaps as scripts ². In this case, we should use absolute imports and pip install -e .

Using this approach allows the files to be run both as a module and as a script on its own. In other words, all the following python utils/something.py, python -m utils.something and import utils.something will work just fine.

---

Add setup.py

setup.py
imptest.py
utils/something.py
utils/other.py

Minimum content for setup.py

from setuptools import setup, find_packages
setup(name='utils', version='0.1', packages=find_packages())

After installing in development mode using pip install -e . you should use absolute import in utils/something.py like you do in imptest.py

import utils.other
...

---

¹ It is best that the codes being tested should reflect the state of codes being used.

² Ok maybe as entry points - but in that case it should have been installed as a package anyway

Answer 3

The problem here is the path, Consider this directory structure

main
 - utils/something.py
 - utils/other.py
 imptest.py

When you try to import other using relative path in to something.py, then you would do something like from . import other. This would work when you execute $ python something.py but would fail when you run $ python imptest.py because in the second scenario it searches for main/other.py which doesn't exist.

So inorder to fix this issue, I would suggest that you write unit tests for something.py & other.py and run them using $ python -m (mod) command. ( I highly recommend this approach )

But.... if you really what your existing code to work without much modification then you can add these 2 lines in something.py file ( this works, but I don't recommend this approach )

import sys, os
sys.path.append(os.getcwd()) # Adding path to this module folder into sys path
import utils.other as other

def do_something():
    print("I am doing something")


def main():
    """
    Main function
    """

    do_something()
    other.do_other()

if __name__ == "__main__":
    main()

Here are some references to get better understanding:

• Unit testing in python
• Absolute vs Relative Imports in python

Answer 4

You need to add an __init__.py file in your main directory so python knows it's a package. Then you could add an import statement like so, from .other import function