Importing Module Between Directories

Question

UPDATE (TO SHOW FINAL CODE)

Since this seemed hard to explain, I shared the project. For those that come to this question, you can see the full project here:

https://github.com/jeffnyman/pacumen

Calling out the files that were problematic for me:

graphical_pacman

https://github.com/jeffnyman/pacumen/blob/master/displays/graphical_pacman.py

layout

https://github.com/jeffnyman/pacumen/blob/master/mechanics/layout.py

With the __init__.py and setup.py files in place, I'm now able to run commands like these:

python displays/graphical_pacman.py

python mechanics/layout.py

All imports now resolve correctly when those commands are executed. You can see all the import statements I use in each file and the placement of the various __init__.py files.

ORIGINAL QUESTION

I cannot get what seems to be a simple thing to work: importing modules between directories. This is all in Python 3 so I don't want to have __init__.py files all over the place if I can help it, which many of the answers here suggest is the "right" way.

I have a structure like this:

project
  displays
    graphical_pacman.py
  mechanics
    layout.py

The layout.py file has a top level function called get_layout() that I want to call from graphical_pacman.py.

Going to the minimum code necessary, in graphical_pacman.py I have:

import layout

if __name__ == '__main__':
  board = layout.get_layout("test_maze.lay")

Shows up fine in IDE, even autocompletes it for me. Running graphical_pacman.py gets me this:

File "displays/graphical_pacman.py", line 3, in <module>
  import layout
ModuleNotFoundError: No module named 'layout'

Then I tried this:

from mechanics.layout import get_layout

if __name__ == '__main__':
  board = mechanics.layout.get_layout("test_maze.lay")

Can't do that either:

File "displays/graphical_pacman.py", line 3, in <module>
  from mechanics.layout import get_layout
ModuleNotFoundError: No module named 'mechanics'

I tried this:

from mechanics import layout

if __name__ == '__main__':
  board = layout.get_layout("test_maze.lay")

I tried this:

from layout import get_layout

if __name__ == '__main__':
  board = get_layout("test_maze.lay")

Doesn't work. Got this:

File "displays/graphical_pacman.py", line 3, in <module>
  from layout import get_layout
ModuleNotFoundError: No module named 'layout'

I tried using relative imports (with the . in front of things) but that also doesn't work. I've also just tried using the * for my import (essentially importing everything). Also doesn't work. When I say "doesn't work" I get some variation of the above errors.

I've tried all of this running the command python graphical_pacman.py from within the displays directory and at the root project directory. The same errors occur each time.

I also tried using a sys.path, such as this sys.path.insert(0, '../mechanics'). I also tried variations on sys.path.append('../') based on other answers I've seen here. Again, all I get are variations on the above errors.

What am I missing?

Answer 1

2nd Update: I have sent a pull request, check it out.
Here's what I did: I created a package out of the project root directory and installed it using pip. Details follow:

Files Added:

pacumen
    __init__.py
    setup.py
    displays
        __init__.py
    library
        __init__.py
    mechanics
         __init__.py

Contents of pacumen/__init__.py:

from . import displays
from . import library
from . import mechanics

Contents of pacumen/setup.py:

import setuptools


setuptools.setup(
    name='pacumen',
    version='0.01dev',
    packages=['displays',
                'library',    
                'mechanics',
            ],
    author='jefferyman',
    author_email='something@something.com',
    description='Pacman. Duh.',
    long_description=open('README.md').read(),        
    )

Contents of pacumen/displays/__init__.py:

from . import graphical_helpers
from . import graphical_pacman
from . import graphical_support

Contents of pacumen/library/__init__.py:

from . import structures
from . import utilities

Contents of pacumen/mechanics/__init__.py:

from . import grid
from . import layout

Changes in files:

pacumen/mechanics/layout.py:

from mechanics.grid import Grid

Make sure your virtual environment is active. (Instructions for that further down).
Finally, navigate to the project root directory and install your project root as a package:

pip install --editable .  # or pip install -e .  (Note the  period at the end)

Now as long as you activate the virtual environment, you should not have any import problems. Do make sure you use the import statements of the style:

from mechanics.grid import Grid

---

Virtual Environment creation and activation:

For those reading, now is a good time to make a virtual environment if one isn't already made. If made make sure to activate it. If not, navigate to the root of the project dir (pacumen) and run

$ python -m venv venvdir

Then, once it is created, run:

<project-root>$ .venvdir\Scripts\activate  # for windows.
OR
<project-root>$ source venvdir/bin/activate  # for linux bash

---

Update: First things to check are that there are no circular imports, which means you are importing something in graphical_pacman from mechanics and something else in mechanics from graphical_pacman. Also make sure your module names don't conflict with built-in python module's names. If it's all good on these fronts,

1. Have you tried this?
   from project.mechanics import layout
2. If that didn't work place __init__.py files in project, displays, mechanics.
   In project/displays/__init__.py add from . import graphical_pacman.
   In project/mechanics/__init__.py add from. import layout.
3. If neither of these worked, create a package out of this, navigate to the package root and install it in you environment with pip install -e .

Answer 2

I have something that works so I have an answer but the full answer is that what I'm doing can't be done in Python, as far as I can tell. Part of the problem here is that as I learned more, I realized my question was not worded well enough in terms of how I wanted things to execute.

Here's how things can work. If in graphical_pacman.py, I do this:

if __name__ == '__main__':
  sys.path.append('../')
  from mechanics import layout

  board = layout.get_layout("test_maze.lay")

This finally works!

BUT ...

I have to run python graphical_pacman.py from within the displays directory rather than at the project root. (Meaning, I can't do this command from the project root: python displays\graphical_pacman.py.)

BUT ...

Execution still runs into an error:

Traceback (most recent call last):
  File "graphical_pacman.py", line 18, in <module>
   from mechanics import layout
  File "../mechanics/layout.py", line 4, in <module>
   from grid import Grid
ModuleNotFoundError: No module named 'grid'

And that's coming from this line in layout.py:

from grid import Grid

If I change that line to a relative import:

from .grid import Grid

NOW my python graphical_pacman.py command works when run from the displays directory.

BUT ...

Now I can't just run python layout.py, which also needs to be possible. That's apparently because the relative import is now in there, and so I get this error:

Traceback (most recent call last):
  File "layout.py", line 4, in <module>
   from .grid import Grid
ModuleNotFoundError: No module named '__main__.grid'; '__main__' is not a package

The problem I'm finding is I need to sometimes run files individually. You'll see mechanics\layout.py has an if __name__ == '__main__': section. So does displays\graphical_support.py. Those are necessary so people can see how things work by running those files directly. But eventually those will be used/imported by a file that's in the root of the directory that is orchestrating everything. And sometimes files (like layout) will depend on others (like grid).

So it seems the answer is that Python can't do what I'm trying to do, which is basically just seek flexibility.

I suppose I could handle all of this by just having all of the files in one directory, which seems to be the best answer here, and seems to be how other projects have done it. That seems like a bad approach but I'm weighing that against fighting with an import mechanism particularly since I've now discovered that the mechanism works in different ways depending on execution.