How to parse multiple nested sub-commands using python argparse?

Question

I am implementing a command line program which has interface like this:

cmd [GLOBAL_OPTIONS] {command [COMMAND_OPTS]} [{command [COMMAND_OPTS]} ...]

I have gone through the argparse documentation. I can implement GLOBAL_OPTIONS as optional argument using add_argument in argparse. And the {command [COMMAND_OPTS]} using Sub-commands.

From the documentation it seems I can have only one sub-command. But as you can see I have to implement one or more sub-commands. What is the best way to parse such command line arguments useing argparse?

Answer 1

I often experience this limitation of argparse as well, it's sometimes important to have parallel, hierarchical subcommands. There are some nice tools out there to help with this (a lot of them in this conversation). However, I felt like many of them still had limitations or required learning a new way of building a parser, so wanted to build another solution. Turns out, a few extensions to argparser make this functionality easy and with only one extra piece to the API.

The trick comes down to building a new parser on demand based on the arguments that are present and any "conditional" that are added by the user. Omitting a few details for clarity, see below.

You can make a new class that extends argument parser to include a method that I called add_conditional(dest, cond, *args, **kwargs) that allows you to add a new argument (with add_argument(*args, **kwargs)) whenever dest==cond. For example:

parser.add_argument("--use-regularization", default=False, action="store_true")
parser.add_conditional("use_regularization", True, "--regularization-lambda", default=1e-3, type=float)

Here's the overall structure of how to do that. First, build a new class that extends ArgumentParser:

class ConditionalArgumentParser(ArgumentParser):
    def __init__(self, *args, **kwargs):
        super(ConditionalArgumentParser, self).__init__(*args, **kwargs)
        self._conditional_parent = []
        self._conditional_condition = []
        self._conditional_args = []
        self._conditional_kwargs = []
        self._num_conditional = 0

Add conditional arguments with a new method:

    def add_conditional(self, dest, cond, *args, **kwargs):
        # attempt to add the conditional argument to a dummy parser to check for errors right away
        _dummy = deepcopy(self)
        _dummy.add_argument(*args, **kwargs)

        # if it passes, store the details to the conditional argument
        assert type(dest) == str, "dest must be a string corresponding to one of the destination attributes"
        self._conditional_parent.append(dest)
        self._conditional_condition.append(self._make_callable(cond))
        self._conditional_args.append(args)
        self._conditional_kwargs.append(kwargs)
        self._num_conditional += 1

Then, overwrite the parse_args method to recursively add and rebuild any conditional arguments as required, then return the arg_parse results from this complete parser:

    def parse_args(self, args=None, namespace=None):
        """Parse the arguments and return the namespace."""
        # if args not provided, use sys.argv
        if args is None:
            args = sys.argv[1:]

        # make a list of booleans to track which conditionals have been added
        already_added = [False for _ in range(self._num_conditional)]

        # prepare the conditionals in a dummy parser so the user can reuse self
        _parser = deepcopy(self)
        _parser = self._prepare_conditionals(_parser, args, already_added)

        # parse the arguments with the conditionals added in the dummy parser
        return ArgumentParser.parse_args(_parser, args=args, namespace=namespace)

I'm omitting some details here, but wanted to just share the basic idea for how this can work. The nice thing about how this is structured is that it's flexible, easy to add conditional arguments, provides useful help messages dependent on other conditionals, and of course only requires the user to learn one new method- add_conditional.

Full disclosure, I wrote a pip-installable package you can download with the above functionality if you want to use it without rewriting it yourself. (I'm an academic researcher at UCL and just wanted to share something I thought was useful).

pip install conditional-parser

Answer 2

The currently most-voted answer's code isn't reproducible.

In the other existing answers, the command line doesn't fail if the user doesn't specify a sub-command. In my use-case, I need the program to fail if no sub-commands are passed (and ideally display the help).

So here is how to do that :

import argparse
import sys


parser = argparse.ArgumentParser(description="My command line.")
subparsers = parser.add_subparsers(title="commands", dest="cmd")

evaluate_parser = subparsers.add_parser("evaluate", help="Run the evaluation using emulated keyboard.")
evaluate_parser.set_defaults(cmd="evaluate")

layout_parser = subparsers.add_parser("show_layout", help="Display the layout over the keyboard for debugging purpose.")
layout_parser.set_defaults(cmd="show_layout")

args = parser.parse_args()

if args.cmd is None:
    parser.print_help(sys.stderr)
    sys.exit(1)

Answer 3

@mgilson has a nice answer to this question. But problem with splitting sys.argv myself is that I lose the nice help message Argparse generates for the user. So I ended up doing this:

import argparse

## This function takes the 'extra' attribute from global namespace and
## re-parses it to create separate namespaces for all other chained commands.
def parse_extra (parser, namespace):
  namespaces = []
  extra = namespace.extra
  while extra:
    n = parser.parse_args(extra)
    extra = n.extra
    namespaces.append(n)

  return namespaces

argparser=argparse.ArgumentParser()
subparsers = argparser.add_subparsers(help='sub-command help', dest='subparser_name')

parser_a = subparsers.add_parser('command_a', help = "command_a help")
## Setup options for parser_a

## Add nargs="*" for zero or more other commands
argparser.add_argument('extra', nargs = "*", help = 'Other commands')

## Do similar stuff for other sub-parsers

Now after first parse all chained commands are stored in extra. I reparse it while it is not empty to get all the chained commands and create separate namespaces for them. And i get nicer usage string that argparse generates.

Answer 4

I came up with the same qustion, and it seems i have got a better answer.

The solution is we shall not simply nest subparser with another subparser, but we can add subparser following with a parser following another subparser.

Here is code to show this:

parent_parser = argparse.ArgumentParser(add_help=False)
parent_parser.add_argument('--user', '-u',
                    default=getpass.getuser(),
                    help='username')
parent_parser.add_argument('--debug', default=False, required=False,
                        action='store_true', dest="debug", help='debug flag')
main_parser = argparse.ArgumentParser()
service_subparsers = main_parser.add_subparsers(title="service",
                    dest="service_command")
service_parser = service_subparsers.add_parser("first", help="first",
                    parents=[parent_parser])
action_subparser = service_parser.add_subparsers(title="action",
                    dest="action_command")
action_parser = action_subparser.add_parser("second", help="second",
                    parents=[parent_parser])

args = main_parser.parse_args()

Answer 5

You could use this multicommand-arg-parser I wrote. It's easy to use, write your template in json format and apply the parser.

https://github.com/antoniofrs/multicommand-args-parser

For example:

[
    {
        "id": "c1_id",
        "command": "command1",
        "help": "Select c1",
        "args": [ ],
        "subCommands": [
            {
                "id": "sc1_id",
                "command": "subcommand1",
                "help": "select sc1",
                "args": [ ]
            },
            {
                "id": "sc2_id",
                "command": "subcommand2",
                "help": "Select c12",
                "args": [ ]
            }
        ]
    },
    {
        "id": "c1_id",
        "command": "command2",
        "help": "Selected c2",
        "args": [ ]
    }
]

Creates the following structure:

command1 --my-args-here
command1 subcommand1 --my-args-here
command1 subcommand2 --my-args-here
command2 --my-args-here

Where id, which you can retrieve from agrs.command_id, will help you to get the command given in input by the user.

You can specify the list of args in the appropriate field (Check the repository readme).

Remember that subcommands inherit command arguments. (e.g subcommand1 contains also the arguments of command)

The json structure is recursive so you can create as many levels of subcommands as you like

Answer 6

In order to parse the sub commands, I used the following (referred from argparse.py code). It parses the sub parser arguments and retains the help for both. Nothing additional passed there.

args, _ = parser.parse_known_args()

Answer 7

I had more or less the same requirements: Being able to set global arguments and being able to chain commands and execute them in order of command line.

I ended up with the following code. I did use some parts of the code from this and other threads.

# argtest.py
import sys
import argparse

def init_args():

    def parse_args_into_namespaces(parser, commands):
        '''
        Split all command arguments (without prefix, like --) in
        own namespaces. Each command accepts extra options for
        configuration.
        Example: `add 2 mul 5 --repeat 3` could be used to a sequencial
                 addition of 2, then multiply with 5 repeated 3 times.
        '''
        class OrderNamespace(argparse.Namespace):
            '''
            Add `command_order` attribute - a list of command
            in order on the command line. This allows sequencial
            processing of arguments.
            '''
            globals = None
            def __init__(self, **kwargs):
                self.command_order = []
                super(OrderNamespace, self).__init__(**kwargs)

            def __setattr__(self, attr, value):
                attr = attr.replace('-', '_')
                if value and attr not in self.command_order:
                    self.command_order.append(attr)
                super(OrderNamespace, self).__setattr__(attr, value)

        # Divide argv by commands
        split_argv = [[]]
        for c in sys.argv[1:]:
            if c in commands.choices:
                split_argv.append([c])
            else:
                split_argv[-1].append(c)

        # Globals arguments without commands
        args = OrderNamespace()
        cmd, args_raw = 'globals', split_argv.pop(0)
        args_parsed = parser.parse_args(args_raw, namespace=OrderNamespace())
        setattr(args, cmd, args_parsed)

        # Split all commands to separate namespace
        pos = 0
        while len(split_argv):
            pos += 1
            cmd, *args_raw = split_argv.pop(0)
            assert cmd[0].isalpha(), 'Command must start with a letter.'
            args_parsed = commands.choices[cmd].parse_args(args_raw, namespace=OrderNamespace())
            setattr(args, f'{cmd}~{pos}', args_parsed)

        return args


    #
    # Supported commands and options
    #
    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)

    parser.add_argument('--print', action='store_true')

    commands = parser.add_subparsers(title='Operation chain')

    cmd1_parser = commands.add_parser('add', formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    cmd1_parser.add_argument('add', help='Add this number.', type=float)
    cmd1_parser.add_argument('-r', '--repeat', help='Repeat this operation N times.',
                                               default=1, type=int)

    cmd2_parser = commands.add_parser('mult', formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    cmd2_parser.add_argument('mult', help='Multiply with this number.', type=float)
    cmd2_parser.add_argument('-r', '--repeat', help='Repeat this operation N times.',
                                               default=1, type=int)

    args = parse_args_into_namespaces(parser, commands)
    return args


#
# DEMO
#

args = init_args()

# print('Parsed arguments:')
# for cmd in args.command_order:
#     namespace = getattr(args, cmd)
#     for option_name in namespace.command_order:
#         option_value = getattr(namespace, option_name)
#         print((cmd, option_name, option_value))

print('Execution:')
result = 0
for cmd in args.command_order:
    namespace = getattr(args, cmd)
    cmd_name, cmd_position = cmd.split('~') if cmd.find('~') > -1 else (cmd, 0)
    if cmd_name == 'globals':
        pass
    elif cmd_name == 'add':
        for r in range(namespace.repeat):
            if args.globals.print:
                print(f'+ {namespace.add}')
            result = result + namespace.add
    elif cmd_name == 'mult':
        for r in range(namespace.repeat):
            if args.globals.print:
                print(f'* {namespace.mult}')
            result = result * namespace.mult
    else:
        raise NotImplementedError(f'Namespace `{cmd}` is not implemented.')
print(10*'-')
print(result)

Below an example:

$ python argstest.py --print add 1 -r 2 mult 5 add 3 mult -r 5 5

Execution:
+ 1.0
+ 1.0
* 5.0
+ 3.0
* 5.0
* 5.0
* 5.0
* 5.0
* 5.0
----------
40625.0

Answer 8

Built a full Python 2/3 example with subparsers, parse_known_args and parse_args (running on IDEone):

from __future__ import print_function

from argparse import ArgumentParser
from random import randint


def main():
    parser = get_parser()

    input_sum_cmd = ['sum_cmd', '--sum']
    input_min_cmd = ['min_cmd', '--min']

    args, rest = parser.parse_known_args(
        # `sum`
        input_sum_cmd +
        ['-a', str(randint(21, 30)),
         '-b', str(randint(51, 80))] +
        # `min`
        input_min_cmd +
        ['-y', str(float(randint(64, 79))),
         '-z', str(float(randint(91, 120)) + .5)]
    )

    print('args:\t ', args,
          '\nrest:\t ', rest, '\n', sep='')

    sum_cmd_result = args.sm((args.a, args.b))
    print(
        'a:\t\t {:02d}\n'.format(args.a),
        'b:\t\t {:02d}\n'.format(args.b),
        'sum_cmd: {:02d}\n'.format(sum_cmd_result), sep='')

    assert rest[0] == 'min_cmd'
    args = parser.parse_args(rest)
    min_cmd_result = args.mn((args.y, args.z))
    print(
        'y:\t\t {:05.2f}\n'.format(args.y),
        'z:\t\t {:05.2f}\n'.format(args.z),
        'min_cmd: {:05.2f}'.format(min_cmd_result), sep='')

def get_parser():
    # create the top-level parser
    parser = ArgumentParser(prog='PROG')
    subparsers = parser.add_subparsers(help='sub-command help')

    # create the parser for the "sum" command
    parser_a = subparsers.add_parser('sum_cmd', help='sum some integers')
    parser_a.add_argument('-a', type=int,
                          help='an integer for the accumulator')
    parser_a.add_argument('-b', type=int,
                          help='an integer for the accumulator')
    parser_a.add_argument('--sum', dest='sm', action='store_const',
                          const=sum, default=max,
                          help='sum the integers (default: find the max)')

    # create the parser for the "min" command
    parser_b = subparsers.add_parser('min_cmd', help='min some integers')
    parser_b.add_argument('-y', type=float,
                          help='an float for the accumulator')
    parser_b.add_argument('-z', type=float,
                          help='an float for the accumulator')
    parser_b.add_argument('--min', dest='mn', action='store_const',
                          const=min, default=0,
                          help='smallest integer (default: 0)')
    return parser


if __name__ == '__main__':
    main()

Answer 9

The solution provide by @Vikas fails for subcommand-specific optional arguments, but the approach is valid. Here is an improved version:

import argparse

# create the top-level parser
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--foo', action='store_true', help='foo help')
subparsers = parser.add_subparsers(help='sub-command help', dest='subparser_name')

# create the parser for the "command_a" command
parser_a = subparsers.add_parser('command_a', help='command_a help')
parser_a.add_argument('bar', type=int, help='bar help')

# create the parser for the "command_b" command
parser_b = subparsers.add_parser('command_b', help='command_b help')
parser_b.add_argument('--baz', choices='XYZ', help='baz help')

# parse some argument lists
argv = ['--foo', 'command_a', '12', 'command_b', '--baz', 'Z']
while argv:
    print(argv)
    options, argv = parser.parse_known_args(argv)
    print(options)
    if not options.subparser_name:
        break

This uses parse_known_args instead of parse_args. parse_args aborts as soon as a argument unknown to the current subparser is encountered, parse_known_args returns them as a second value in the returned tuple. In this approach, the remaining arguments are fed again to the parser. So for each command, a new Namespace is created.

Note that in this basic example, all global options are added to the first options Namespace only, not to the subsequent Namespaces.

This approach works fine for most situations, but has three important limitations:

• It is not possible to use the same optional argument for different subcommands, like myprog.py command_a --foo=bar command_b --foo=bar.
• It is not possible to use any variable length positional arguments with subcommands (nargs='?' or nargs='+' or nargs='*').
• Any known argument is parsed, without 'breaking' at the new command. E.g. in PROG --foo command_b command_a --baz Z 12 with the above code, --baz Z will be consumed by command_b, not by command_a.

These limitations are a direct limitation of argparse. Here is a simple example that shows the limitations of argparse -even when using a single subcommand-:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('spam', nargs='?')
subparsers = parser.add_subparsers(help='sub-command help', dest='subparser_name')

# create the parser for the "command_a" command
parser_a = subparsers.add_parser('command_a', help='command_a help')
parser_a.add_argument('bar', type=int, help='bar help')

# create the parser for the "command_b" command
parser_b = subparsers.add_parser('command_b', help='command_b help')

options = parser.parse_args('command_a 42'.split())
print(options)

This will raise the error: argument subparser_name: invalid choice: '42' (choose from 'command_a', 'command_b').

The cause is that the internal method argparse.ArgParser._parse_known_args() it is too greedy and assumes that command_a is the value of the optional spam argument. In particular, when 'splitting' up optional and positional arguments, _parse_known_args() does not look at the names of the arugments (like command_a or command_b), but merely where they occur in the argument list. It also assumes that any subcommand will consume all remaining arguments. This limitation of argparse also prevents a proper implementation of multi-command subparsers. This unfortunately means that a proper implementation requires a full rewrite of the argparse.ArgParser._parse_known_args() method, which is 200+ lines of code.

Given these limitation, it may be an options to simply revert to a single multiple-choice argument instead of subcommands:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--bar', type=int, help='bar help')
parser.add_argument('commands', nargs='*', metavar='COMMAND',
                 choices=['command_a', 'command_b'])

options = parser.parse_args('--bar 2 command_a command_b'.split())
print(options)
#options = parser.parse_args(['--help'])

It is even possible to list the different commands in the usage information, see my answer https://stackoverflow.com/a/49999185/428542

Answer 10

Another package which supports parallel parsers is "declarative_parser".

import argparse
from declarative_parser import Parser, Argument

supported_formats = ['png', 'jpeg', 'gif']

class InputParser(Parser):
    path = Argument(type=argparse.FileType('rb'), optional=False)
    format = Argument(default='png', choices=supported_formats)

class OutputParser(Parser):
    format = Argument(default='jpeg', choices=supported_formats)

class ImageConverter(Parser):
    description = 'This app converts images'

    verbose = Argument(action='store_true')
    input = InputParser()
    output = OutputParser()

parser = ImageConverter()

commands = '--verbose input image.jpeg --format jpeg output --format gif'.split()

namespace = parser.parse_args(commands)

and namespace becomes:

Namespace(
    input=Namespace(format='jpeg', path=<_io.BufferedReader name='image.jpeg'>),
    output=Namespace(format='gif'),
    verbose=True
)

Disclaimer: I am the author. Requires Python 3.6. To install use:

pip3 install declarative_parser

Here is the documentation and here is the repo on GitHub.

Answer 11

Improving on the answer by @mgilson, I wrote a small parsing method which splits argv into parts and puts values of arguments of commands into hierarchy of namespaces:

import sys
import argparse


def parse_args(parser, commands):
    # Divide argv by commands
    split_argv = [[]]
    for c in sys.argv[1:]:
        if c in commands.choices:
            split_argv.append([c])
        else:
            split_argv[-1].append(c)
    # Initialize namespace
    args = argparse.Namespace()
    for c in commands.choices:
        setattr(args, c, None)
    # Parse each command
    parser.parse_args(split_argv[0], namespace=args)  # Without command
    for argv in split_argv[1:]:  # Commands
        n = argparse.Namespace()
        setattr(args, argv[0], n)
        parser.parse_args(argv, namespace=n)
    return args


parser = argparse.ArgumentParser()
commands = parser.add_subparsers(title='sub-commands')

cmd1_parser = commands.add_parser('cmd1')
cmd1_parser.add_argument('--foo')

cmd2_parser = commands.add_parser('cmd2')
cmd2_parser.add_argument('--foo')

cmd2_parser = commands.add_parser('cmd3')
cmd2_parser.add_argument('--foo')


args = parse_args(parser, commands)
print(args)

It behaves properly, providing nice argparse help:

For ./test.py --help:

usage: test.py [-h] {cmd1,cmd2,cmd3} ...

optional arguments:
  -h, --help        show this help message and exit

sub-commands:
  {cmd1,cmd2,cmd3}

For ./test.py cmd1 --help:

usage: test.py cmd1 [-h] [--foo FOO]

optional arguments:
  -h, --help  show this help message and exit
  --foo FOO

And creates a hierarchy of namespaces containing the argument values:

./test.py cmd1 --foo 3 cmd3 --foo 4
Namespace(cmd1=Namespace(foo='3'), cmd2=None, cmd3=Namespace(foo='4'))

Answer 12

You could try arghandler. This is an extension to argparse with explicit support for subcommands.

Answer 13

parse_known_args returns a Namespace and a list of unknown strings. This is similar to the extra in the checked answer.

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
sub = parser.add_subparsers()
for i in range(1,4):
    sp = sub.add_parser('cmd%i'%i)
    sp.add_argument('--foo%i'%i) # optionals have to be distinct

rest = '--foo 0 cmd2 --foo2 2 cmd3 --foo3 3 cmd1 --foo1 1'.split() # or sys.argv
args = argparse.Namespace()
while rest:
    args,rest =  parser.parse_known_args(rest,namespace=args)
    print args, rest

produces:

Namespace(foo='0', foo2='2') ['cmd3', '--foo3', '3', 'cmd1', '--foo1', '1']
Namespace(foo='0', foo2='2', foo3='3') ['cmd1', '--foo1', '1']
Namespace(foo='0', foo1='1', foo2='2', foo3='3') []

An alternative loop would give each subparser its own namespace. This allows overlap in positionals names.

argslist = []
while rest:
    args,rest =  parser.parse_known_args(rest)
    argslist.append(args)

Answer 14

You can always split up the command-line yourself (split sys.argv on your command names), and then only pass the portion corresponding to the particular command to parse_args -- You can even use the same Namespace using the namespace keyword if you want.

Grouping the commandline is easy with itertools.groupby:

import sys
import itertools
import argparse    

mycommands=['cmd1','cmd2','cmd3']

def groupargs(arg,currentarg=[None]):
    if(arg in mycommands):currentarg[0]=arg
    return currentarg[0]

commandlines=[list(args) for cmd,args in intertools.groupby(sys.argv,groupargs)]

#setup parser here...
parser=argparse.ArgumentParser()
#...

namespace=argparse.Namespace()
for cmdline in commandlines:
    parser.parse_args(cmdline,namespace=namespace)

#Now do something with namespace...

untested

Answer 15

you can use the package optparse

import optparse
parser = optparse.OptionParser()
parser.add_option("-f", dest="filename", help="corpus filename")
parser.add_option("--alpha", dest="alpha", type="float", help="parameter alpha", default=0.5)
(options, args) = parser.parse_args()
fname = options.filename
alpha = options.alpha