Find out which arguments were passed explicitly in argparse

Question

I need to handle two ways of configuring an application. One is via command line arguments, and the other one is from a JSON file (actually, the JSON file is the result of storing the arguments from a previous run).

I can handle this by merging the two namespace objects, something like this:

cli_args = some_parser.parse_args()
with open(json_file, 'r') as f:
    json_args = json.load(f)
all_args = argparse.Namespace()
all_args.__dict__ = {**vars(cli_args), **json_args}
# or all_args.__dict__ = {**json_args, **vars(cli_args)}

The problem is on the last line. If I choose the first version, arguments from the JSON file take precedence. If I choose the second version, arguments from the CLI take precedence.

I would like to have CLI arguments take precedence, but only if they were actually specified. The problem arises when the parser admits default values. In this case, the cli_args object will be populated with default values, and this will take precedence over JSON arguments.

For a simple example, let me take this parser:

parser = argparse.ArgumentParser()
parser.add_argument('--foo', default='FOO')
parser.add_argument('--bar', default='BAR')

Say I have a JSON file with

{
  "foo": "myfoo",
  "bar": "mybar"
}

and that I invoke my application with python myapp.py --foo hello.

I would like to get a namespace object having foo=hello, bar=mybar. Both ways of merging the arguments will give something different. First, if I give the JSON file precedence I will obtain foo=myfoo, bar=mybar. If I give the CLI the precedence, I get foo=hello, bar=BAR.

The problem is that I cannot see a way to distinguish which arguments in the namespace returned from parser.parse_args() were populated by the user, and which ones were filled in using default settings.

Is there a way to ask argparse which arguments were actually explicitly set on the command line, as opposed to being filled with defaults?

Answer 1

Here's my extension of the answer by @Andrea. Also uses parse_known_args to get the config file from the CLI, as well.

import argparse
import tomllib

cfg_parser = argparse.ArgumentParser()
cfg_parser.add_argument("--config")

parser = argparse.ArgumentParser(description="Hi")
parser.add_argument("--foo", default=0, type=int)
parser.add_argument("--bar", default=0, type=int)

cfg_args, rem_strs = cfg_parser.parse_known_args()

if cfg_args.config is not None:
    with open(cfg_args.config, "rb") as f:
        cfg_namespace = argparse.Namespace(**tomllib.load(f))
else:
    cfg_namespace = None

args = parser.parse_args(rem_strs, namespace=cfg_namespace)

Answer 2

Elliot's answer got me going. But, it fails in at least the case where an argument's action is "count" (what is _Sentinel() + 1?). The following works in at least that case.

Basically, make a deep copy of the parser object, then set the default for all options (at least, those seen on the command line) to None, then re-parse the command line and look at what comes out the other end un-None.

import argparse
from argparse import Namespace
import copy

parser = argparse.ArgumentParser()
parser.add_argument('--goo', action='store_false', default=True)
parser.add_argument('--foo', action='count', default=10)
args = parser.parse_args()

# now, (deep) copy the *parser*
p2 = copy.deepcopy(parser)
# and, set all defaults of options seen on command line to None
p2.set_defaults(**{x:None for x in vars(args)})
# and, parse the same arguments with the new parser
got = p2.parse_args()

# now, what is not None is explicit (use Argparse Namespace)
explicit = Namespace(**{key:(value is not None) for key, value in vars(got).items()})

print("args.foo:", args.foo)
print("explicit.foo:", explicit.foo)
print("explicit.goo:", explicit.goo)

running that without --foo gives

args.foo: 10
explicit.foo: False
explicit.goo: False

running that with --foo gives

args.foo: 11
explicit.foo: True
explicit.goo: False

Answer 3

I have a solution that detects if an argument was explicitly set on the command line or if its value is from the default. Here is an example that accepts the login of a user:

import argparse                                                                 
import os                                                                       
                                                                                
parser = argparse.ArgumentParser()
login = os.getlogin()
parser.add_argument("-l",
                    "--login",
                    type=str,
                    default=login)
args = parser.parse_args()
print(f"login: {args.login}, default={id(login) == id(args.login)}")

If the user specifies a login the parser will store a new value and the id of the default and the value in the result namespace will be different. If nothing was specified the namespace value will have the same id as the default. I am unsure if relying on such undocumented implementation features is a good idea, but for my version of Python (3.11.2) is works.

Answer 4

For the next person here looking for a solution to this problem, we can use Andrea's workaround and a sentinel to find out exactly which arguments were passed explicitly.

As Andrea pointed out, if we pass a namespace to parse_args(), then any existing names in the namespace override the defaults of the parser. So if we use a namespace that contains all the parameter names, then any that have changed their value after parsing must have been changed by explicit arguments.

We need to set everything in that namespace to something different than what the parser might use as a default anyway. The parser's defaults could well be None so that doesn't work; instead we need a sentinel object.

import argparse
from argparse import Namespace

# Create a sentinel.
# Could use sentinel = object() instead; this way makes it clear 
# if we ever print it that the object is a sentinel.
class _Sentinel:
    pass
sentinel = _Sentinel()

parser = argparse.ArgumentParser()
parser.add_argument('--foo', type=int, default=10)
args = parser.parse_args()

# Make a copy of args where everything is the sentinel.
sentinel_ns = Namespace(**{key:sentinel for key in vars(args)})
parser.parse_args(namespace=sentinel_ns)

# Now everything in sentinel_ns that is still the sentinel was not explicitly passed.
explicit = Namespace(**{key:(value is not sentinel)
                for key, value in vars(sentinel_ns).items()})

print("args.foo:", args.foo)
print("explicit.foo:", explicit.foo)

Running this script with no arguments will print

args.foo: 10
explicit.foo: False

And running this script with --foo 10 will print

args.foo: 10
explicit.foo: True

Answer 5

I could not find a way to ask argparse what arguments were explicitly set, but I found an alternative way to solve my issue.

Namely, the parse_args method accept a Namespace object that it will populate with the parsing results. Hence I can read the JSON content into a Namespace, then use parse_args() to add the arguments from command line. This will override the JSON settings, but only for explictly set arguments, not for defaults:

json_args = argparse.Namespace()
with open(json_file, 'r') as f:
    json_args.__dict__ = json.load(f)
all_args = parser.parse_args(namespace=json_args)