How to "fold" python files used as modules into the main script file?

Question

Suppose I have two Python script files: foo and utils/bar.py in some directory. In foo, I have:

import os
import sys
sys.path.append(os.path.dirname(os.path.realpath(__file__)))
from utils.bar import func1, func2

# ... code using func1 or func2
# ... perhaps some code using utils.bar.func3

that is, the subdirectory utils is added to the module search path, and then utils/bar.py is used as the source of utils.bar functions.

I'm not much of a pythonista, so I'm not sure if this hack is customary or not, but regardless - I need a single deployable script file, not a hierarchy of files. So, I want to "fold" the file hierarchy all into a single file - put the contents of bar.py into foo in some way, so that the script will continue working even if I then removed the utils/ subdirectory and the bar.py file.

I could remove all references to utils.bar and just copy the plain functions from bar.py into foo; but I was wondering if there was something less "brute-force" than that.

(There are actually multiple files in multiple subdirectories, I just gave a simplified example.)

Answer 1

The zipapp Module

If you want to bundle your app into a single runnable file then you can use the standard library module zipapp to create a zip archive that the python binary knows how to execute, or optionally make the archive itself runnable.

If you have a project structured like:

└── myapp/
    ├── foo.py  # main script
    └── utils/
        ├── __init__.py
        └── bar.py

And where foo.py has an entrypoint function eg.

from util.bar import greet

def main():
    print(greet(who='world'))

if '__main__' == __name__:
    main()

Then you can create a runnable archive like so:

cd path/to/dir/containing/myapp
python -m zipapp myapp --main foo:main --output myapp.pyz

Here --main foo:main means upon starting the app, import a module called foo and execute a function in it called main().

You can then run the archive like so:

python myapp.pyz

If you want to make the archive runnable (only works for unix-based systems), then you can also add the --python flag. This sets the shebang (#!) for the archive. Example:

python -m zipapp myapp \
    --main foo:main    \
    --output myapp.pyz \
    --python '/usr/bin/env python3.10'
# You can now run the archive like so:
./myapp.pyz

By default, files in the archive are stored uncompressed. You can compress the files by using the --compress flag.

Using Third Party Packages

zipapp only includes the files in the source directory. If you have third party packages you need to use, then you will either need to setup a virtual environment on the target machine or use pip to directory install these packages into your source directory before you package it. Example:

pip install --target path/to/myapp requests

To stop bloating your source directory with installed packages, you may find it easier to create a build directory where you copy your source to and install your desired packages when building the zip archive.

Distributing Libraries Separately

If the size of the third party libraries is large, it may become problematic to constantly redistribute these libraries with every change you make to your own source code. You can instead distribute them once as a zip file, and use PYTHONPATH to tell python to make the libraries in the zip file available to your application. To package the requests package in a separate zip file and have it be importable from your own code you would do:

python -m pip install requests --target build
pushd build  # Temporarily cd to build dir
             # This is required for files to be packaged in the zip file correctly
python -m zipfile --create ../lib.zip .
popd

You would run your app like so:

PYTHONPATH=lib.zip python -m zipapp myapp.pyz

NB: Be sure to build both your app zip file and your library zip file separately and in isolation. You do not want the app zip file to accidentally include library code, or vice versa.

Accessing Resources

If you have config files or other data files you want to package up with the archive, then you will need to access them a different way. This is because they will no longer be directly accessible on the file system. That is open('path/to/file') will not work as you want it to. Instead you will need to use the importlib.resources module, and to have your data files in python package.

As example, suppose you have a file called config.toml you want to package with your archive. You would need to add it to your project like so:

└── myapp/
    ├── foo.py  # main script
    ├── utils/
    │   ├── __init__.py
    │   └── bar.py
    └── data/            # The file needs to be contained by a package.
        ├── __init__.py  # A proper package, not just a namespace package.
        └── config.toml

You would then load the contents of the file like so:

from importlib.resources import read_text

text = read_text('data', 'config.toml')

NB. This form will also work when running your app as a standard script. So NO need for constructions like:

if is_archive():  # pseudo-function
    text = importlib.resources.read_text('data', 'config.toml')
else: # if script
    with open('data/config.toml') as f:
        text = f.read()

Answer 2

I would suggest to structure your project like this.

.
├── app                -> The main package, which contains all source code files
│   ├── __init__.py    -> This file must exist so that app is treated as a package
│   ├── utils          -> The utils package
│   │   ├── __init__.py
│   │   └── bar.py
│   └── main.py        -> The entry point of your code
├── tests              -> Contains all unit tests
│   └── __init__.py
├── .gitignore
└── README.md

Make sure that you have __init.py__ file inside the app and utils directory to turn them into packages.

In your main.py:

import os
import sys
from app.utils.bar import func1, func2

if __name__ == '__main__':
    print('Start my program')
    func1()
    func2()

To start your program, make sure that you stay at the root of the project, then run:

$ python -m app.main

Note that the program is executed as a module, not a script. This answer has the difference well-explained.

Answer 3

Assuming this is an import issue.

You are treating this as a full module (With a __init__.py & all of that.), but it isn't. You are trying to import a single file, which can be done using the 'sys' module:

# importing sys
from bar.py import func1, func2
import sys

# adding utils path
sys.path.insert(0, 'Path/To/Utils') # Must be absolute path, I recommend using os.getcwd()


# Code Using func1 & func2

Source: Geeks For Geeks (Click the link to see the specific post.)