CODEWITHSUNDEEP
pythonpython-importpython-packagingrelative-import
Marco Necci
Marco Necci

Reputation: 329

Python: weird import behaviour

I have a package with the following structure

projectX
├── data 
├── results
└── projectX
    ├── stats
    │   ├── __init__.py
    │   ├── stats_worker.py
    │   └── stats_wroker2.py
    ├── __init__.py
    ├── main_worker.py
    └── interface.py

Notice that the project folder and the source folder share the same name (projectX).

The problem is a bit complex and needs some premises to be done

Premises:

I always launch the interface and always have launched the interface. No other script is called

My problem first arose when trying to import main_worker from stats_worker2.

I couldn't make it work as a relative import. So I tried to append the parent directory to the sys.path, both from stats_worker2 and from the interface (separately). I tried various ways that I don't remember.

It finally worked by adding the following lines to stats_worker2

sys.path.append('../projectX')
from projectX import main_worker

Now the weirdness starts

After calling the script a couple of times (I don't remember exactly how many), the above sintax doesn't work anymore. Because there is an: ImportError: No module named projectX

EDIT: the above part in italic was due to the fact that I was missing an __init__.py in the projectX source folder. All other symptoms are still there.

But, weird enough, I can now import successfully the main worker from the stats_worker2 with:

import main_worker

Without any directory being appended to sys.path!

However, Sphinx doesn't work anymore! Specifically, it fails to import all of the modules of the projectX package.

Bonus weirdness

I used to successfully import stats_worker from the interface, now I still can import it, but my IDE (Pycharm) marks that import as unused. It is actually used and if I comment the import, the interface crashes as it is expected to (I already have invalidated Pycharm cache)

Upvotes: 1

Views: 712

Answers (1)

wim
wim

Reputation: 363304

  1. Add an empty __init__.py file into the source root†.
  2. Your runtime environment will need to be setup so that the project root†† is visible in sys.path. The project root should not have an __init__.py. It isn't necessary for the project directory name to match the package name, but that doesn't really harm anything either.
  3. Remove any sys.path.append stuff from python source files, and don't ever do that again.
  4. from projectX import main_worker is a correct import statement for the stats_worker2.py module. Don't use import main_worker.

For step 2, the usual way to do this is via "installing" your package, with python setup.py develop. i.e. you need to write a setup.py. Since you apparently don't have a setup.py, as a cheap workaround you can export PYTHONPATH=<absolute_path_to_project_root> instead.

†Source root is the deeper projectX dir, it's the same directory which contains the main_worker.py module.

††Project root is the shallow projectX dir, it's the same directory which contains data and results subdirectories.

Upvotes: 1

Related Questions