gunicorn with compiled python source

Question

I have compiled my python files including gunicorn with PyInstaller on centOS 7. I'm trying to run those executables on another centOS 7 machine I'm getting ModuleNotFoundError: No module named 'manage' but my module manage is present in the same directory in executable form. Here is the full command and its output.

./gunicorn -c /opt/myproject/configuration/gunicorn_conf.ini --bind unix:/etc/myproject/myproject.sock -m 007 manage:app --preload

!!!
!!! WARNING: configuration file should have a valid Python extension.
!!!

Traceback (most recent call last):
  File "gunicorn", line 8, in <module>
  File "site-packages/gunicorn/app/wsgiapp.py", line 58, in run
  File "site-packages/gunicorn/app/base.py", line 228, in run
  File "site-packages/gunicorn/app/base.py", line 72, in run
  File "site-packages/gunicorn/arbiter.py", line 58, in __init__
  File "site-packages/gunicorn/arbiter.py", line 118, in setup
  File "site-packages/gunicorn/app/base.py", line 67, in wsgi
  File "site-packages/gunicorn/app/wsgiapp.py", line 49, in load
  File "site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
  File "site-packages/gunicorn/util.py", line 358, in import_app
  File "importlib/__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'manage'
[20316] Failed to execute script gunicorn

Am I missing something here? or gunicorn does not work with compiled files? if it does not, then is there any alternative for it other than using the source?

Answer 1

Errors like Import module errors and ModuleNotFoundError: No module named occurs when PyInstaller can't detect some python modules.

Those are known as Hidden Imports.

=> See answer from Stephen Collins above on how to fix this.

Here are some troubleshooting information:

• You can also use pyinstaller --debug=imports to list hidden imports (doc)
• avoid unexpected runtime error by reviewing the file created by pyInstaller that list delayed and conditional dependencies:

$ cat warn-app.txt
This file lists modules PyInstaller was not able to find. This does not
necessarily mean this module is required for running your program. Python and
Python 3rd-party packages include a lot of conditional or optional modules. For
example the module 'ntpath' only exists on Windows, whereas the module
'posixpath' only exists on Posix systems.

Types if import:
* top-level: imported at the top-level - look at these first
* conditional: imported within an if-statement
* delayed: imported within a function
* optional: imported within a try-except-statement

IMPORTANT: Do NOT post this list to the issue-tracker. Use it as a basis for
            tracking down the missing module yourself. Thanks!

missing module named org - imported by copy (optional)
missing module named 'org.python' - imported by pickle (optional), xml.sax (delayed, conditional)
missing module named winreg - imported by importlib._bootstrap_external (conditional), mimetypes (optional), urllib.request (delayed, conditional, optional), platform (delayed, optional), werkzeug.debug (delayed, conditional)
missing module named nt - imported by os (delayed, conditional, optional), ntpath (optional), shutil (conditional), importlib._bootstrap_external (conditional), pathlib (conditional), ctypes (delayed, conditional)

Answer 2

I had an issue similar where the import failed for gunicorn.glogger. I solved it by adding the import to the top of the module explicitly.

from gunicorn import glogger

I am using gunicorn 20.0.4 and I do not see a module called manage. Is it possible this is a dependency from another library?

Another possible Fix

Hidden imports are specified in the pyinstaller docs, you can use the switch --hidden-import in your build command to include implicitly used libraries within the binary. If you have multiple imports you can use a : to separate the list.

Invoking GUnicorn

Green unicorn is built to be started from the command line. This is easy to do within a docker container or by running the command on a host machine. Unfortunately this is not the case when using pyinstaller as you need a "python" entry point. I had to build a custom application from the gunicorn docs (copy/paste) to allow my application to start up as a script. From here I was able to tell pyinstaller to use this script to start the server under if __name__ == '__main__':

See example below.

Bugs

Even with the binary built with required dependencies I ran into an issue where the web application would stop responding after a random number of requests. I did not investigate further as it pushed me into another direction of building an RPM for easy installation.

Update February 4th, 2021

Setting arguments to GUnicorn.

import gunicorn.app.base


class StandaloneApplication(gunicorn.app.base.BaseApplication):

    def __init__(self, app, options=None):
        self.options = options or {}
        self.application = app
        super().__init__()

    def load_config(self):
        config = {key: value for key, value in self.options.items()
                  if key in self.cfg.settings and value is not None}
        for key, value in config.items():
            self.cfg.set(key.lower(), value)

    def load(self):
        return self.application


if __name__ == '__main__':
    options = {
        'bind': 'unix:/etc/myproject/myproject.sock',
        'preload': True,
        'config': '/opt/myproject/configuration/gunicorn_conf.ini',
        'umask': '007',
        '<argument name>': '<argument value>',
    }
    StandaloneApplication(<your flask application object or factory>, options).run()

Haven't tested this specific setup but I am using something very similar. Each of the GUnicorn switch/option in the CLI maps directly to python variables, you can find that here. These options are set in our options dictionary which the server will accept as the second augument in the StandaloneApplication constructor. You will need to pass in the app object as the first argument or application factory function from your manage module.

I do recommend looking at python configuration files. You can also export these as environment variables and use os.getenv("<env name>") to access them. It just keeps the server from using hardcoded values that other users will have to change in the source.