Local imports work in bundled PyInstaller app but in Python source

Question

This issue has plagued me for the last few months, I need a more experienced opinion. We have a CLI Python application that uses a gRPC server to communicate with other backend services. Its structured something like this:

app
    - gRPC_service
        - __init__.py
        - service_related_files.py
        - service_tests
            - __init__.py
            - test_service.py
    
    - src
        - __init__.py
        - src_files.py

    - python-win
        - python37.dll

    - gRPC_service.spec

A few notes about the application:

• The src directory houses the lower level machine learning code. The gRPC_service directory acts as a wrapper around the src code and sends processed requests to a given client

• The python-win directory is a specific version of the Python Interpreter. Many things throughout the code base are reliant on a specific version of Python (3.7.9). This was our solution to making the code base a bit more portable. A developer can clone the repository and immediately have the necessary version of Python installed along with the plethora of 3rd party dependencies after running a Create-VirtualEnvironment.ps1 script that uses the python.exe in the python-win directory.

The number one issue I have faced when developing this application is namespace and importing issues, and I'm not exactly sure what's causing it. We have a sub-application within the src directory that only imports modules from within the src package and uses 3rd party libraries. This works just fine with no ModuleNotFound errors.

Issues begin to surface when importing src modules from within the gRPC_service package. Even though both packages have __init__.py files, ModuleNotFound errors will be thrown if the PYTHONPATH is not modified at runtime. A work-around solution to this is to collect all the different file paths to each package within app and add them to sys.path. It goes without saying this is inconvenient.

This works for importing a majority of the modules, but to add to the confusion, some of the modules from the src package can only be imported after modifying sys.path AND adding a src. prefix to all of the local imports within the src package. Adding this prefix to local imports breaks the sub-application in the src package that I was speaking of earlier. A 'src can't be found' error gets thrown in the sub-app when doing this.

Additionally, without adding the src. prefixes to corresponding imports, no ModuleNotFound errors are thrown when the gRPC_service is bundled as a PyInstaller .exe. I have modified the pathex within the .spec file. The app works just fine when bundled - how do I get equivalent behavior when just running Python from source?

So I am looking for some advice from Python devs who have worked on large code bases. Is this issue common? What can I do to alleviate the inconvenience of modifying the PYTHONPATH at runtime? Is there a fix-all solution that tends to the needs of both applications within this codebase?

Answer 1

I can think of two solutions,

1. Split gRPC_services into its own project and repository and add it as a dependency of this project and install and import it like any other third party library.

2. move the gRPC_services folder inside of the src folder as a subpackage.