Include submodules on click

Question

I am trying to make a kind of recursive call on my first Click CLI app. The main point is to have sub-commands associated to the first and, so, I was trying to separate it all in different files/modules to improve it's maintainability.

I have the current directory:

root
|-commands
|-project
|---__init__
|---command1
|---command2
|-database
|---__init__
|---command1
|---command2

This is my main file:

import click
from commands.project import project
from commands.database import database


@click.group(help="Main command")
def main():
    pass


main.add_command(project)
main.add_command(database)

My projects __init__ file:

from commands.project.command1 import *
from commands.project.command2 import *
import click


@click.group(help="Projects")
def project():
    pass


project.add_command(command1)
project.add_command(command2)

My commands.project.command1 file:

import click


@click.command()
def command1():
    """
    Execute all the steps required to update the project.
    """
    pass

The main point here is that, every time I want to add a new subcommand, I need to:

1. Add .py file with all code to the command, in respective subcommand/submodule folder (obviously!)

2. Add it's import statement on it's __init__ file

3. Relate this new command to it's parent (project/database, in this case)

Is there any way to do a circular/dynamic load to avoid step no.2 and 3?

---

EDIT

After tried Stephen Rauch way, it successfully includes all provided files, but none of the commands works with - only with function name (eg: update-project -> update_project).

root
|-commands
|-project
|---update
|---install_project
|-database
|---command_one
|---command_two

main.py

# main command ----------------------------------------------------------- ###
@click.group(help="CLI tool!", context_settings=dict(max_content_width=120))
def main():
    pass


# PROJECT command group -------------------------------------------------------- ###
@main.group(cls=group_from_folder("commands/project"),
            short_help="Project installation and upgrade utils.",
            help="Project installation and upgrade.")
def project():
    pass

commands/project/install_project.py

import click    

@click.command(name="install-project",
               help="This options allows you to easily install project",
               short_help="Install a brand new project")
@click.pass_context
def install_project(ctx):

CLI result main project --help (note the install_project instead install-project sub command)

Usage: main project [OPTIONS] COMMAND [ARGS]...

  Project installation and upgrade.

Options:
  --help  Show this message and exit.

Commands:
  install_project                     Install a brand new project one

Answer 1

I suggest you just read your commands from specific Python package and then add to you entry group.

Suppose we have such structure:

|--app
   |--commands
      |--__init__.py
      |--group1
         |--__init__.py
         |--command1.py
      |--group2
         |--__init__.py
         |--command2.py
|--__init__.py
|--cli.py

Then your commands files need to contain one click.Command with a specified name and a function with a name 'command':

import click

@click.command(name="your-first-command")
def command():
    pass

Init files in each of your group need to contain doc string to have proper 'help' value for your click.Group.

And most interesting cli.py:

import click
import importlib
import pkgutil
import os.path


def get_commands_from_pkg(pkg) -> dict:
    pkg_obj = importlib.import_module(pkg)

    pkg_path = os.path.dirname(pkg_obj.__file__)

    commands = {}
    for module in pkgutil.iter_modules([pkg_path]):
        module_obj = importlib.import_module(f"{pkg}.{module.name}")
        if not module.ispkg:
            commands[module_obj.command.name] = module_obj.command

        else:
            commands[module.name.replace('_', '-')] = click.Group(
                context_settings={'help_option_names': ['-h', '--help']},
                help=module_obj.__doc__,
                commands=get_commands_from_pkg(f"{pkg}.{module.name}")
            )

    return commands


@click.group(context_settings={'help_option_names': ['-h', '--help']}, help="Your CLI",
             commands=get_commands_from_pkg('app.commands'))
def cli():
    pass

As you can see we recursively create click groups and add the click command to the specific group.

Answer 2

Modifying the example from here, you can eliminate steps two and three. I suggest creating a custom class for each folder via a closure. This completely eliminates the need for the __init__.py in the commands folder. Additionally there is no need to import the folder (module) or the commands in the folder.

Custom Group Class Creator:

import click
import os

def group_from_folder(group_folder_name):

    folder = os.path.join(os.path.dirname(__file__), group_folder_name)

    class FolderCommands(click.MultiCommand):

        def list_commands(self, ctx):
            return sorted(
                f[:-3] for f in os.listdir(folder) if f.endswith('.py'))

        def get_command(self, ctx, name):
            namespace = {}
            command_file = os.path.join(folder, name + '.py')
            with open(command_file) as f:
                code = compile(f.read(), command_file, 'exec')
                eval(code, namespace, namespace)
            return namespace[name.replace('-', '_').lower()]

    return FolderCommands

Using the Custom Class:

To use the custom class, first place the commands (as structured in the question) into a folder. Then decorate the group command using the cls parameter, and pass a custom class which was initialized pointing to the folder containing the commands.

@cli.group(cls=group_from_folder('project'))
def group():
    "command for grouping"

Test Code:

@click.group()
def cli():
    "My awesome script"


@cli.group(cls=group_from_folder('group'))
def group():
    "command for grouping"


if __name__ == "__main__":
    commands = (
        'group command-test',
        'group',
        'group --help',
        '',
    )

    import sys, time

    time.sleep(1)
    print('Click Version: {}'.format(click.__version__))
    print('Python Version: {}'.format(sys.version))
    for cmd in commands:
        try:
            time.sleep(0.1)
            print('-----------')
            print('> ' + cmd)
            time.sleep(0.1)
            cli(cmd.split())

        except BaseException as exc:
            if str(exc) != '0' and \
                    not isinstance(exc, (click.ClickException, SystemExit)):
                raise

File group/command-test.py

import click


@click.command('command-test')
def command_test():
    """
    Execute all the steps required to update the project.
    """
    click.echo('Command Test')

Results:

Click Version: 6.7
Python Version: 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
-----------
> group command-test
Command Test
-----------
> group
Usage: test.py group [OPTIONS] COMMAND [ARGS]...

  command for grouping

Options:
  --help  Show this message and exit.

Commands:
  command-test  Execute all the steps required to update the...
-----------
> group --help
Usage: test.py group [OPTIONS] COMMAND [ARGS]...

  command for grouping

Options:
  --help  Show this message and exit.

Commands:
  command-test  Execute all the steps required to update the...
-----------
> 
Usage: test.py [OPTIONS] COMMAND [ARGS]...

  My awesome script

Options:
  --help  Show this message and exit.

Commands:
  group  command for grouping