relative paths for modules in python

Question

I've attempted a few different techniques trying to do something that to me seems doable but I guess I am missing some gotchas about python (using 2.7 but would like this to work also for 3.* if possible).

I am not sure about terminology like package or module, but to me the following seems quite a "simple" doable scenario.

This is the directory structure:

.
├── job
│   └── the_script.py
└── modules
    ├── __init__.py
    └── print_module.py

The content of the_script.py:

# this does not work
import importlib
print_module = importlib.import_module('.print_module', '..modules')

# this also does not work
from ..modules import print_module

print_module.do_stuff()

The content of print_module:

def do_stuff():
    print("This should be in stdout")

I would like to run all this "relative paths" stuff as:

/job$ python2 the_script.py

But the importlib.import_module gives various errors:

• if I just use 1 input parameter ..modules.print_module, then I get: TypeError("relative imports require the 'package' argument")
• if I use 2 input parameters (as in the example above), then I get: ValueError: Empty module name

On the other hand using the from ..modules syntax I get: ValueError: Attempted relative import in non-package.

I think the __init__.py empty file should be enough to qualify that code as "packages" (or modules? not sure about the terminology), but it seems there's something I am missing about how to manage relative paths.

I read that in the past people was hacking this using the path and other functions from import os and import sys, but according to the official docs (python 2.7 and 3.*) this should not be needed anymore.

What am I doing wrong and how could I achieve the result of printing the content modules/print_module.do_stuff calling it from a script in the "relative directory" job/?

Answer 1

I found a solution using sys and os.

The script the_script.py should be:

import sys
import os
lib_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../modules'))
sys.path.append(lib_path)

# commenting out the following shows the `modules` directory in the path
# print(sys.path)

import print_module

print_module.do_stuff()

Then I can run it via command line no matter where I am in the path e.g.:

• /job$ python2 the_script.py
• <...>/job$ python2 <...>/job/the_script.py

Answer 2

If you follow the structure of this guide here: http://docs.python-guide.org/en/latest/writing/structure/#test-suite (highly recommend reading it all, it is very helpful) you will see this:

To give the individual tests import context, create a tests/context.py file:

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

import sample

Then, within the individual test modules, import the module like so:

from .context import sample

This will always work as expected, regardless of installation method.

Translated in your case this means:

root_folder
├── job
│   ├── context.py <- create this file
│   └── the_script.py
└── modules
    ├── __init__.py
    └── print_module.py

In the context.py file write the lines shown above, but import modules instead of import samples

Finally in your the_script.py: from .context import module and you will be set to go!

Good luck :)

Answer 3

If you are not sure about terminology go to very nice tutorials:

http://docs.python-guide.org/en/latest/writing/structure/#modules

and

http://docs.python-guide.org/en/latest/writing/structure/#packages

But for your structure:

.
├── job
│   └── the_script.py
└── modules
    ├── __init__.py
    └── print_module.py

just say in the the_script.py:

import sys
sys.append('..')
import modules.print_module

This will add parent directory to PYTHONPATH, and python will see directory 'parallel' to job directory and it will work.

I think that at the most basic level it is sufficent to know that:

1. package is any directory with __init__.py file
2. module is a file with .py, but when you are importing module you omit extension.