No module error in Python 3.6 with Click library

Question

I'm trying to build a CLI in python by using the package click. The Python version I'm using is 3.6

This is the main of my application:

import os
import click

cmd_folder = os.path.join(os.path.dirname(__file__), 'commands')


class IAMCLI(click.MultiCommand):

    def list_commands(self, ctx):
        rv = []
        for filename in os.listdir(cmd_folder):
            if filename.endswith('.py') and \
                    filename.startswith('cmd_'):
                rv.append(filename[4:-3])
        rv.sort()
        return rv

    def get_command(self, ctx, cmd_name):
        ns = {}
        fn = os.path.join(cmd_folder, 'cmd_{}.py'.format(cmd_name))
        with open(fn) as f:
            code = compile(f.read(), fn, 'exec')
            eval(code, ns, ns)
        return ns['cli']


@click.command(cls=IAMCLI)
@click.option('--env', default='dev', type=click.Choice(['dev', 'staging', 'production']),
              help='AWS Environment')
@click.pass_context
def cli():
    """AWS IAM roles and policies management CLI."""
    pass


if __name__ == '__main__':
    cli()

and this is the tree:

├── cli
│   ├── __init__.py
│   ├── aws
│   │   ├── __init__.py
│   │   ├── policy.py
│   │   └── role.py
│   ├── cli.py
│   └── commands
│       ├── __init__.py
│       └── cmd_dump.py

the cmd_dump.py looks like this:

import click

from cli.aws.role import fetch_roles


@click.command('dump', short_help='Dump IAM resources')
@click.pass_context
def cli():
  pass

the problem is that when I try to run python cli/cli.py --help this is what I get:

File "cli/commands/cmd_dump.py", line 3, in <module>
    from cli.aws.role import fetch_roles
ModuleNotFoundError: No module named 'cli.aws'; 'cli' is not a package

Any idea about that?

Answer 1

I will try to give another answer based on my approach when starting development of a new python project. Do you plan to distribute your project, or maybe just share it with someone? If you do, what do you think - will this someone be happy with needing to remember the command

$ python path/to/project/codebase/cli/cli.py --help

to use your tool? Wouldn't it be easier for him to remember the command

$ cli --help

instead?

I suggest you to start with packaging of your project right away - write a minimal setup script:

from setuptools import setup, find_packages

setup(
    name='mypkg',
    version='0.1',
    packages=find_packages(),
    install_requires=['click'],
    entry_points={
        'console_scripts': ['cli=cli.cli:cli'],
    },
)

You can always enhance your setup script when new requirements emerge. Place the setup script in the root directory of your codebase:

├── setup.py
├── cli
│   ├── __init__.py
│   ├── aws
...

Now run python setup.py develop or even better, pip install --editable=.¹ from the codebase root directory (where the setup.pyscript is). You have installed your project in the development mode and now can invoke

$ cli --help

with all the imports being resolved correctly (which would resolve your issue). But you gain much more besides that - you gain a way to package your project to be ready to be distributed to your target users and a clean command line interface which your users will invoke the same way as you just did.

Now continue with the project development. If you change the code of the cli command, it will be applied on the fly so you don't need to reinstall the project each time you change anything.

Once you are ready with the project development and want to deliver it to your users, issue:

$ python setup.py bdist_wheel

This will package your project into a installable wheel file (you will need to install wheel package to be able to invoke the command: pip install wheel --user). Usually it will reside in the dist subdirectory of the codebase root dir. Give this file to the user. To install the file, he will issue

$ pip install Downloads/mypkg-0.1-py3-none.whl --user

and can start tp use your tool right away:

$ cli --help

This is a very much simplified description and there is a lot of stuff to learn, but there is also a ton of helpful materials to guide you through the process.

If you want to learn more about the topic: as a quickstart reference, I would recommend the excellent PyPA packaging guide. For packaging click commands, their own docs are more than sufficient.

---

1. I would encourage you to use pip for distribution and packaging development where applicable as it is a standard tool for that.

Answer 2

Do not run scripts inside packages! Packages are made to be imported in code but not to run scripts inside them. The rest is non related with import errors.

For example:

├── cli # package
│   ├── __init__.py
│   ├── aws
│   │   ├── __init__.py
│   │   ├── policy.py
│   │   └── role.py
│   ├── cli.py
│   │   └── commands
│   │       ├── __init__.py
│   │       └── cmd_dump.py 
├── run_this_module.py

Module to execute run_this_module.py:

import cli

"""Do your code here"""

Answer 3

I have done this a hundred times. It is tempting to name a package the same thing at multiple levels, but resist! I tend to have __main__.py in my packages these days, which helps solve the problem that you are having.

I suspect you are having a namespace issue. Try renaming your package or the internal file called cli.py. Be sure to refactor any imports that are attempting to use these.

Also, change your function name to something besides cli. Same issue.

├── cli                    <- rename this to 'my_app' or something else
│   ├── __init__.py
│   ├── aws
│   │   ├── __init__.py
│   │   ├── policy.py
│   │   └── role.py
│   ├── cli.py             <- or maybe it is easier to rename this
│   └── commands
│       ├── __init__.py
│       └── cmd_dump.py