Add top level argparse arguments after subparser args

Question

How can you allow for top-level program arguments to be added after using a subcommand from a subparser?

I have a program that includes several subparsers to allow for subcommands, changing the behavior of the program. Here is an example of how its set up:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import argparse


def task_a():
    print('did task_a')

def task_c():
    print('did task_c')

def task_d():
    print('did task_d')


def run_foo(args):
    a_arg = args.a
    c_arg = args.c
    if a_arg:
        task_a()
    if c_arg:
        task_c()


def run_bar(args):
    a_arg = args.a
    d_arg = args.d
    if a_arg:
        task_a()
    if d_arg:
        task_d()

def parse():
    '''
    Run the program
    arg parsing goes here, if program was run as a script
    '''
    # create the top-level parser
    parser = argparse.ArgumentParser()
    # add top-level args
    parser.add_argument("-a", default = False, action = "store_true", dest = 'a')

    # add subparsers
    subparsers = parser.add_subparsers(title='subcommands', description='valid subcommands', help='additional help', dest='subparsers')

    # create the parser for the "foo" command
    parser_foo = subparsers.add_parser('foo')
    parser_foo.set_defaults(func = run_foo)
    parser_foo.add_argument("-c", default = False, action = "store_true", dest = 'c')

    # create the parser for the "bar" downstream command
    parser_bar = subparsers.add_parser('bar')
    parser_bar.set_defaults(func = run_bar)
    parser_bar.add_argument("-d", default = False, action = "store_true", dest = 'd')

    # parse the args and run the default parser function
    args = parser.parse_args()
    args.func(args)

if __name__ == "__main__":
    parse()

When I run the program I can call a subcommand with its args like this:

$ ./subparser_order.py bar -d
did task_d

$ ./subparser_order.py foo -c
did task_c

But if I want to include the args from the top level, I have to call it like this:

$ ./subparser_order.py -a foo -c
did task_a
did task_c

However, I think this is confusing, especially if there are many top-level args and many subcommand args; the subcommand foo is sandwiched in the middle here and harder to discern.

I would rather be able to call the program like subparser_order.py foo -c -a, but this does not work:

$ ./subparser_order.py foo -c -a
usage: subparser_order.py [-h] [-a] {foo,bar} ...
subparser_order.py: error: unrecognized arguments: -a

In fact, you cannot call the top-level args at all after specifying a subcommand:

$ ./subparser_order.py foo -a
usage: subparser_order.py [-h] [-a] {foo,bar} ...
subparser_order.py: error: unrecognized arguments: -a

Is there a solution that will allow for the top-level args to be included after the subcommand?

Answer 1

There is actually a way to do it. You can use parse_known_args, take the namespace and unparsed arguments and pass these back to a parse_args call. It will combine and override in the 2nd pass and any left over arguments from there on will still throw parser errors.

Simple example, here is the setup:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action='store_true')
sp = parser.add_subparsers(dest='subargs')
sp_1 = sp.add_parser('foo')
sp_1.add_argument('-b', action='store_true')
print(parser.parse_args())

In the proper order for argparse to work:

- $ python3 argparse_multipass.py
Namespace(a=False, subargs=None)
- $ python3 argparse_multipass.py -a
Namespace(a=True, subargs=None)
- $ python3 argparse_multipass.py -a foo
Namespace(a=True, b=False, subargs='foo')
- $ python3 argparse_multipass.py foo
Namespace(a=False, b=False, subargs='foo')
- $ python3 argparse_multipass.py foo -b
Namespace(a=False, b=True, subargs='foo')
- $ python3 argparse_multipass.py -a foo -b
Namespace(a=True, b=True, subargs='foo')

Now, you can't parse arguments after a subparser kicks in:

- $ python3 argparse_multipass.py foo -b -a
usage: argparse_multipass.py [-h] [-a] {foo} ...
argparse_multipass.py: error: unrecognized arguments: -a

However, you can do a multi-pass to get your arguments back. Here is the updated code:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action='store_true')
sp = parser.add_subparsers(dest='subargs')
sp_1 = sp.add_parser('foo')
sp_1.add_argument('-b', action='store_true')
args = parser.parse_known_args()
print('Pass 1: ', args)
args = parser.parse_args(args[1], args[0])
print('Pass 2: ', args)

And the results for it:

- $ python3 argparse_multipass.py
Pass 1:  (Namespace(a=False, subargs=None), [])
Pass 2:  Namespace(a=False, subargs=None)
- $ python3 argparse_multipass.py -a
Pass 1:  (Namespace(a=True, subargs=None), [])
Pass 2:  Namespace(a=True, subargs=None)
- $ python3 argparse_multipass.py -a foo
Pass 1:  (Namespace(a=True, b=False, subargs='foo'), [])
Pass 2:  Namespace(a=True, b=False, subargs='foo')
- $ python3 argparse_multipass.py foo
Pass 1:  (Namespace(a=False, b=False, subargs='foo'), [])
Pass 2:  Namespace(a=False, b=False, subargs='foo')
- $ python3 argparse_multipass.py foo -b
Pass 1:  (Namespace(a=False, b=True, subargs='foo'), [])
Pass 2:  Namespace(a=False, b=True, subargs='foo')
- $ python3 argparse_multipass.py -a foo -b
Pass 1:  (Namespace(a=True, b=True, subargs='foo'), [])
Pass 2:  Namespace(a=True, b=True, subargs='foo')
- $ python3 argparse_multipass.py foo -b -a
Pass 1:  (Namespace(a=False, b=True, subargs='foo'), ['-a'])
Pass 2:  Namespace(a=True, b=True, subargs='foo')

This will maintain original functionality but allow continued parsing for when subparsers kick in. Additionally you could make disordered parsing out of the thing entirely if you do something like this:

args, unknown_args = parser.parse_known_args()
while len(unknown_args):
    args, unknown_args = parser.parse_known_args(unknown_args, args)

It will keep parsing arguments in case they are out of order until it has completed parsing all of them. Keep in mind this one will keep going if there is an unknown argument that stays in there. Mind want to add something like this:

    previously_unknown_args = None
    args, unknown_args = parser.parse_known_args()
    while len(unknown_args) and unknown_args != previously_unknown_args:
        args, unknown_args = parser.parse_known_args(unknown_args, args)
        previously_unknown_args = unknown_args 
    args = parser.parse_args(unknown_args, args)

Just keep a previously unparsed arguments object and compare it, when it breaks do a final parse_args call to make the parser run its own errors path.

It's not the most elegant solution but I ran into the exact same problem where my arguments on the main parser were used as optional flags additionally on top of what was specified in a sub parser.

Keep the following in mind though: This code will make it so a person can specify multiple subparsers and their options in a run, the code that these arguments invoke should be able to deal with that.

Answer 2

Once the top level parser encounters 'foo' it delegates parsing to parser_foo. That modifies the args namespace, and returns. The top level parser does not resume parsing. It just handles any errors returned by the subparser.

In [143]: parser=argparse.ArgumentParser()
In [144]: parser.add_argument('-a', action='store_true');
In [145]: sp = parser.add_subparsers(dest='cmd')
In [146]: sp1 = sp.add_parser('foo')
In [147]: sp1.add_argument('-c', action='store_true');

In [148]: parser.parse_args('-a foo -c'.split())
Out[148]: Namespace(a=True, c=True, cmd='foo')

In [149]: parser.parse_args('foo -c'.split())
Out[149]: Namespace(a=False, c=True, cmd='foo')

In [150]: parser.parse_args('foo -c -a'.split())
usage: ipython3 [-h] [-a] {foo} ...
ipython3: error: unrecognized arguments: -a

You can keep it from choking on the unrecognized argument, but it won't resume parsing:

In [151]: parser.parse_known_args('foo -c -a'.split())
Out[151]: (Namespace(a=False, c=True, cmd='foo'), ['-a'])

You could also add an argument with the same flag/dest to the subparser.

In [153]: sp1.add_argument('-a', action='store_true')
In [154]: parser.parse_args('foo -c -a'.split())
Out[154]: Namespace(a=True, c=True, cmd='foo')

but the default for the sub entry overrides the toplevel value (there has been bug/issue discussion over this behavior).

In [155]: parser.parse_args('-a foo -c'.split())
Out[155]: Namespace(a=False, c=True, cmd='foo')

It might be possible to parse that extra string with a two stage parser, or with a custom _SubParsersAction class. But with the argparse as it is, there isn't an easy way around this behavior.