CODEWITHSUNDEEP
pythonconfiguration-filesstring-interpolation
Anaphory
Anaphory

Reputation: 6422

Use value from previous config file in interpolation

I am using a program which uses python's configparser.ConfigParser to work with config files as instructions to build a thing. It is set up so that multiple command line file arguments can be specified, and specifications in later files override things set in earlier files.

This means I can set up a basic model in a trivial.ini file

[admin]
basename = trivial_model
[model basic]
data = data.tsv

and extend it with a more complex model extension.ini

[admin]
basename = trivial_model_extended
[model basic]
model = bsvs

and program trivial.ini extension.ini will behave as if it had got

[admin]
basename = trivial_model_extended
[model basic]
data = data.tsv
model = bsvs

Now it would be nice to write this somewhat more modular, to be able to combine multiple such extensions without caring about details too much, giving each a different base file name. I thought maybe this might work

[admin]
basename = %(basename)s_extended
[model basic]
model = bsvs

but with the current implementation, I get configparser.InterpolationDepthError: Recursion limit exceeded in value substitution: option 'basename' in section 'admin' contains an interpolation key which cannot be substituted in 10 steps. Raw value: '%(basename)s_extended'.

Is there an easy, builtin, or elegant way to enable incremental specifications like this, either through changing the current implementation (which seems to boil down to

parser.add_argument(
    "config",
    nargs="+")

args = parser.parse_args()
c = configparser.ConfigParser()
for conf in args.config:
    c.read(conf)

) or through some clever [default] sections or values in the config files (or both if necessary)?

Upvotes: 3

Views: 2536

Answers (2)

Anaphory
Anaphory

Reputation: 6422

Evaluation at get-time

Recursive interpolation in the way you are looking for is not possible out-of-the-box because interpolation only acts at the time when ConfigParser.get(section, option) is called, not at the time the configuration file is read. From BasicInterpolation's docstring:

[...]  All reference expansions are done late, on demand. [...]

How do we fix this?

The _read method of ConfigParser does calls self._interpolation.before_read, but by that time it has already overwritten the old internal values of the ConfigParser object, so even

class BasicReadInterpolation (configparser.BasicInterpolation):
    def before_read(self, parser, section, option, value):
        L = []
        import pdb
        pdb.set_trace()
        interpolations = parser[parser.default_section]
        interpolations.update(parser[section])
        self._interpolate_some(
            parser, option, L, value, section, interpolations, 1)
        return ''.join(L)

will not do the trick all by itself.

You will also have to overload ConfigParser._read, the method which contains most of the parsing magic, and ConfigParser._join_multiline_values, which currently contains the call to self._interpolation.before_read. (Which to me is a silly place to put it.)

Upvotes: 0

Hai Vu
Hai Vu

Reputation: 40773

A few comments:

  • You cannot recurse the basename definition like you did above. My approach is to have a [DEFAULT] section with something other than basename, for example, the trivial.ini might look like this:

    [DEFAULT]
basename_default = default from trivial.ini

[admin]
basename = trivial_model

[model basic]
data = data.tsv

  • Note that the [DEFAULT] section needs to be all uppercase

  • Next, I might have an additiona .ini file, which I call more.ini and it looks like this:

    [admin]
basename = %(basename_default)s and more

  • Also, you don't need a loop to read the config files: just give the read() method a list of filenames where the later file will overwrite the first.

Putting it together:

parser = argparse.ArgumentParser()
parser.add_argument("config", nargs="+")
args = parser.parse_args('trivial.ini extension.ini more.ini'.split())

cfg = ConfigParser.ConfigParser()
cfg.read(args.config)

admin = 'admin'
model_basic = 'model basic'

print('basename:', cfg.get(admin, 'basename'))
print('defaults:', cfg.defaults())

The output:

basename: default from trivial.ini and more
defaults: OrderedDict([('basename_default', 'default from trivial.ini')])

Upvotes: 1

Related Questions