With python click, how can I avoid duplicating argument code used by multiple subcommands

Question

I have a group of subcommands that all operate on a list of URLs that can optionally be passed as an argument. How can I assign this argument to the group instead to avoid duplicating the argument definition on every subcommand?

Current code:

from config import site_list

@click.group()
def cli():
    pass

@cli.command()
@cli.argument('sites', nargs=-1)
def subcommand_one():
    if sites:
        site_list = sites
    etc...

@cli.command()
@cli.argument('sites', nargs=-1)
def subcommand_two():
    if sites:
        site_list = sites
    etc...

Example invocation:

$ python sites.py subcommand_one www.example.com www.example2.com

I tried moving the argument decorator to the group like this:

@click.group()
@click.argument('sites', nargs=-1)
def cli(sites):
    if sites:
        site_list = sites

But then I would get this error:

$ python sites.py subcommand_one
Usage: sites.py [OPTIONS] [SITES] COMMAND [ARGS]...
Try "sites.py --help" for help.

Error: Missing command.

Answer 1

I think there is an actual solution supported by Click using the @click.pass_context.

When you want to define a group of commands which all share for example a common argument and a common option, then you can define them on the group level and add them to a context object like described in the Click documentation.

@click.group(chain=True)
@click.argument("dataset_directory", type=click.Path(exists=True))
@click.option("-s", "--split-names", help="The splits to preprocess.", required=True,
              default=["trainset", "devset", "testset"], show_default=True)
@click.pass_context
def cli(ctx, dataset_directory, split_names):
    """
    Prepare the dataset for training

    DATASET_DIRECTORY The absolute path to the data directory.
    """
    ctx.ensure_object(dict)
    ctx.obj["DIRECTORY"] = dataset_directory
    ctx.obj["SPLITS"] = split_names

Then the individual commands of this group can get passed the context and use the values from the context object instead of defining their own arguments and options.

@cli.command("create")
@click.pass_context
def create(ctx):
    create_semantics_json_from_csv(ctx.obj["DIRECTORY"], ctx.obj["SPLITS"])


@cli.command("tokenize")
@click.pass_context
def tokenize(ctx):
    preprocess_tokenize_semantics_json(ctx.obj["DIRECTORY"], ctx.obj["SPLITS"])

The invocation of the commands is then possible like:

my-cli-app /path/to/data create tokenize

Answer 2

If there is a specific nargs = -1 argument that you would like to decorate only onto the group, but be applicable to all commands as needed, you can do that with some of extra plumbing like:

_{This answer is inspired by this answer.}

Custom Class

class GroupNArgsForCommands(click.Group):
    """Add special arguments on group"""

    def __init__(self, *args, **kwargs):
        super(GroupNArgsForCommands, self).__init__(*args, **kwargs)
        cls = GroupNArgsForCommands.CommandArgument

        # gather the special arguments for later
        self._cmd_args = {
            a.name: a for a in self.params if isinstance(a, cls)}

        # strip out the special arguments from self
        self.params = [a for a in self.params if not isinstance(a, cls)]

    class CommandArgument(click.Argument):
        """class to allow us to find our special arguments"""

    @staticmethod
    def command_argument(*param_decls, **attrs):
        """turn argument type into type we can find later"""

        assert 'cls' not in attrs, "Not designed for custom arguments"
        attrs['cls'] = GroupNArgsForCommands.CommandArgument

        def decorator(f):
            click.argument(*param_decls, **attrs)(f)
            return f

        return decorator

    def group(self, *args, **kwargs):
        # any derived groups need to be the same type
        kwargs['cls'] = GroupNArgsForCommands

        def decorator(f):
            grp = super(GroupNArgsForCommands, self).group(
                *args, **kwargs)(f)
            self.add_command(grp)

            # any sub commands need to hook the same special args
            grp._cmd_args = self._cmd_args

            return grp

        return decorator

    def add_command(self, cmd, name=None):

        # call original add_command
        super(GroupNArgsForCommands, self).add_command(cmd, name)

        # if this command's callback has desired parameters add them
        import inspect
        args = inspect.signature(cmd.callback)
        if len(args.parameters):
            for arg_name in reversed(list(args.parameters)):
                if arg_name in self._cmd_args:
                    cmd.params[:] = [self._cmd_args[arg_name]] + cmd.params

Using the Custom Class:

To use the custom class, pass the cls parameter to the click.group() decorator, use the @GroupNArgsForCommands.command_argument decorator for the special argument, and then add a parameter of the same name as the special argument to any commands as needed.

@click.group(cls=GroupNArgsForCommands)
@GroupNArgsForCommands.command_argument('special', nargs=-1)
def a_group():
    """My project description"""

@a_group.command()
def a_command(special):
    """a command under the group"""

How does this work?

This works because click is a well designed OO framework. The @click.group() decorator usually instantiates a click.Group object but allows this behavior to be over ridden with the cls parameter.
So it is a relatively easy matter to inherit from click.Group in our own class and over ride desired methods.

In this case we over ride click.Group.add_command() so that when a command is added we can examine the command callback parameters to see if they have the same name as any of our special arguments. If they match, the argument is added to the command's arguments just as if it had been decorated directly.

In addition GroupNArgsForCommands implements a command_argument() method. This method is used as a decorator when adding the special argument instead of using click.argument()

Test Class

import click

@click.group(cls=GroupNArgsForCommands)
@GroupNArgsForCommands.command_argument('sites', nargs=-1)
def cli():
    click.echo("cli group")

@cli.command()
def command_one(sites):
    click.echo("command_one: {}".format(sites))

@cli.group()
def subcommand():
    click.echo("subcommand group")

@subcommand.command()
def one():
    click.echo("subcommand_one")

@subcommand.command()
def two(sites):
    click.echo("subcommand_two: {}".format(sites))

if __name__ == "__main__":
    commands = (
        'command_one site1 site2',
        'command_one site1',
        'command_one',
        'subcommand',
        'subcommand one site1 site2',
        'subcommand one site1',
        'subcommand one',
        'subcommand two site1 site2',
        'subcommand two site1',
        'subcommand two',
        '--help',
        'command_one --help',
        'subcommand --help',
        'subcommand one --help',
        'subcommand two --help',
        '',
    )

    import sys, time

    time.sleep(1)
    print('Click Version: {}'.format(click.__version__))
    print('Python Version: {}'.format(sys.version))
    for command in commands:
        try:
            time.sleep(0.1)
            print('-----------')
            print('> ' + command)
            time.sleep(0.1)
            cli(command.split())

        except BaseException as exc:
            if str(exc) != '0' and \
                    not isinstance(exc, (click.ClickException, SystemExit)):
                raise

Results:

Click Version: 6.7
Python Version: 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
-----------
> command_one site1 site2
cli group
command_one: ('site1', 'site2')
-----------
> command_one site1
cli group
command_one: ('site1',)
-----------
> command_one
cli group
command_one: ()
-----------
> subcommand
cli group
Usage: test.py subcommand [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  one
  two
-----------
> subcommand one site1 site2
Usage: test.py subcommand one [OPTIONS]

Error: Got unexpected extra arguments (site1 site2)
cli group
subcommand group
-----------
> subcommand one site1
cli group
subcommand group
Usage: test.py subcommand one [OPTIONS]

Error: Got unexpected extra argument (site1)
-----------
> subcommand one
cli group
subcommand group
subcommand_one
-----------
> subcommand two site1 site2
cli group
subcommand group
subcommand_two: ('site1', 'site2')
-----------
> subcommand two site1
cli group
subcommand group
subcommand_two: ('site1',)
-----------
> subcommand two
cli group
subcommand group
subcommand_two: ()
-----------
> --help
Usage: test.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  command_one
  subcommand
-----------
> command_one --help
cli group
Usage: test.py command_one [OPTIONS] [SITES]...

Options:
  --help  Show this message and exit.
-----------
> subcommand --help
cli group
Usage: test.py subcommand [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  one
  two
-----------
> subcommand one --help
cli group
subcommand group
Usage: test.py subcommand one [OPTIONS]

Options:
  --help  Show this message and exit.
-----------
> subcommand two --help
cli group
subcommand group
Usage: test.py subcommand two [OPTIONS] [SITES]...

Options:
  --help  Show this message and exit.
-----------
> 
Usage: test.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  command_one
  subcommand

Answer 3

click.argument just returns a decorator like any other, so you can assign it to some variable:

import click

@click.group()
def cli():
    pass

sites_argument = click.argument('sites', nargs=-1)

@cli.command()
@sites_argument
def subcommand_one(sites):
    ...

@cli.command()
@sites_argument
def subcommand_two(sites):
    ...