How would you save a simple settings/configuration file in Python?

Question

I don't care if it's JSON, pickle, YAML, or whatever.

All other implementations I have seen are not forward compatible, so if I have a configuration file, add a new key in the code, and then load that configuration file, it'll just crash.

Are there a simple way to do this?

Answer 1

If you want to use something like an INI file to hold settings, consider using configparser which loads key value pairs from a text file, and can easily write back to the file.

INI file has the format:

[Section]
key = value
key with spaces = somevalue

Answer 2

Configuration files in Python

There are several ways to do this depending on the file format required.

ConfigParser [.ini format]

I would use the standard configparser approach unless there were compelling reasons to use a different format.

Write a file like so:

# Python 2.x
# from ConfigParser import SafeConfigParser
# config = SafeConfigParser()

# Python 3.x
from configparser import ConfigParser
config = ConfigParser()

config.read('config.ini')
config.add_section('main')
config.set('main', 'key1', 'value1')
config.set('main', 'key2', 'value2')
config.set('main', 'key3', 'value3')

with open('config.ini', 'w') as f:
    config.write(f)

The file format is very simple with sections marked out in square brackets:

[main]
key1 = value1
key2 = value2
key3 = value3

Values can be extracted from the file like so:

# Python 2.x
# from ConfigParser import SafeConfigParser
# config = SafeConfigParser()

# Python 3.x
from configparser import ConfigParser
config = ConfigParser()

config.read('config.ini')

print(config.get('main', 'key1')) # -> "value1"
print(config.get('main', 'key2')) # -> "value2"
print(config.get('main', 'key3')) # -> "value3"

# getfloat() raises an exception if the value is not a float
a_float = config.getfloat('main', 'a_float')

# getint() and getboolean() also do this for their respective types
an_int = config.getint('main', 'an_int')

JSON [.json format]

JSON data can be very complex and has the advantage of being highly portable.

Write data to a file:

import json

config = {"key1": "value1", "key2": "value2"}

with open('config1.json', 'w') as f:
    json.dump(config, f)

Read data from a file:

import json

with open('config.json', 'r') as f:
    config = json.load(f)

# Edit the data
config['key3'] = 'value3'

# Write it back to the file
with open('config.json', 'w') as f:
    json.dump(config, f)

YAML

A basic YAML example is provided in this answer. More details can be found on the pyYAML website.

Answer 3

I was faced with the same problem, but in addition I’d like to read configuration variables from hard coded fields in case if the configuration file doesn't exist.

My variant:

import json

class Configurator:

    def __init__(self):
        # Hard coded values if config file doesn't exist
        self.alpha: int = 42
        self.bravo: float = 3.14
        self.charlie: str = "8.8.8.8"
        self.delta: list = ["Lorem", "ipsum", "dolor", "sit", "amet"]
        self.echo: dict = {"Winter": "is coming"}

    def read_config_file(self, config_file_name: str = "config.json"):
        try:
            with open(config_file_name) as conf_file:
                for k, v in json.loads(conf_file.read()).items():
                    setattr(self, k, v)
        except Exception as e:
            print(f"Error was detected while reading {config_file_name}: {str(e)}. Hard coded values will be applied")

    def save_config_file(self, config_file_name: str = "config.json"):
        try:
            conf_items = {k: v for k, v in vars(self).items() if isinstance(v, (int, float, str, list, dict))}
            with open(config_file_name, "w") as conf_file:
                json.dump(conf_items, conf_file, sort_keys=False, indent=2)
        except Exception as e:
            print(f"Error was detected while saving {config_file_name}: {str(e)}")

from configurator import Configurator

if __name__ == '__main__':
    conf = Configurator()

    # Read the configuration (values from file or hard coded values if file doesn't exist)
    conf.read_config_file()

    # Using values from configuration
    a = conf.alpha

    # Changing values in the configuration
    conf.bravo += 1

    # Save changed configuration to the file
    conf.save_config_file()

If the configuration file doesn't exist, it appears after first call of conf.save_config_file(). If you change config.json after that, variables from file must "beat" hard coded variables at the next time.

The code is a little hacky, test it before using.

Answer 4

For a simple configuration file, I prefer a JSON file, e.g., file conf.json:

{
  "version": 1,
  "bind": {
    "address": "127.0.0.1",
    "port": 8080
  },
  "data": {
    "a": [1, 2, 3],
    "b": 2.5
  }
}

Then create this custom JSON configuration reader:

import json

class Dict(dict):
    """dot.notation access to dictionary attributes"""
    __getattr__ = dict.__getitem__
    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

class Config(object):
    @staticmethod
    def __load__(data):
        if type(data) is dict:
            return Config.load_dict(data)
        elif type(data) is list:
            return Config.load_list(data)
        else:
            return data

    @staticmethod
    def load_dict(data: dict):
        result = Dict()
        for key, value in data.items():
            result[key] = Config.__load__(value)
        return result

    @staticmethod
    def load_list(data: list):
        result = [Config.__load__(item) for item in data]
        return result

    @staticmethod
    def load_json(path: str):
        with open(path, "r") as f:
            result = Config.__load__(json.loads(f.read()))
        return result

Lastly, load it using command:

conf = Configuration.load_json('conf.json')

Now you can access your configuration using a dot, ".", e.g.:

print(conf.version)
print(conf.bind.address)
print(conf.bind.port)
print(conf.data.a)
print(conf.data.b)

Answer 5

Try using cfg4py:

1. Hierarchical design, multiple environments supported, so never mess up development settings with production site settings.
2. Code completion. Cfg4py will convert your YAML content into a Python class, and then code completion is available while you type your code.
3. many more...

Disclaimer: I'm the author of this module

Answer 6

ConfigParser Basic example

The file can be loaded and used like this:

#!/usr/bin/env python

import ConfigParser
import io

# Load the configuration file
with open("config.yml") as f:
    sample_config = f.read()
config = ConfigParser.RawConfigParser(allow_no_value=True)
config.readfp(io.BytesIO(sample_config))

# List all contents
print("List all contents")
for section in config.sections():
    print("Section: %s" % section)
    for options in config.options(section):
        print("x %s:::%s:::%s" % (options,
                                  config.get(section, options),
                                  str(type(options))))

# Print some contents
print("\nPrint some contents")
print(config.get('other', 'use_anonymous'))  # Just get the value
print(config.getboolean('other', 'use_anonymous'))  # You know the datatype?

which outputs

List all contents
Section: mysql
x host:::localhost:::<type 'str'>
x user:::root:::<type 'str'>
x passwd:::my secret password:::<type 'str'>
x db:::write-math:::<type 'str'>
Section: other
x preprocessing_queue:::["preprocessing.scale_and_center",
"preprocessing.dot_reduction",
"preprocessing.connect_lines"]:::<type 'str'>
x use_anonymous:::yes:::<type 'str'>

Print some contents
yes
True

As you can see, you can use a standard data format that is easy to read and write. Methods like getboolean and getint allow you to get the datatype instead of a simple string.

Writing configuration

import os
configfile_name = "config.yaml"

# Check if there is already a configuration file
if not os.path.isfile(configfile_name):
    # Create the configuration file as it doesn't exist yet
    cfgfile = open(configfile_name, 'w')

    # Add content to the file
    Config = ConfigParser.ConfigParser()
    Config.add_section('mysql')
    Config.set('mysql', 'host', 'localhost')
    Config.set('mysql', 'user', 'root')
    Config.set('mysql', 'passwd', 'my secret password')
    Config.set('mysql', 'db', 'write-math')
    Config.add_section('other')
    Config.set('other',
               'preprocessing_queue',
               ['preprocessing.scale_and_center',
                'preprocessing.dot_reduction',
                'preprocessing.connect_lines'])
    Config.set('other', 'use_anonymous', True)
    Config.write(cfgfile)
    cfgfile.close()

results in

[mysql]
host = localhost
user = root
passwd = my secret password
db = write-math

[other]
preprocessing_queue = ['preprocessing.scale_and_center', 'preprocessing.dot_reduction', 'preprocessing.connect_lines']
use_anonymous = True

XML Basic example

Seems not to be used at all for configuration files by the Python community. However, parsing / writing XML is easy and there are plenty of possibilities to do so with Python. One is Beautiful Soup:

from BeautifulSoup import BeautifulSoup

with open("config.xml") as f:
    content = f.read()

y = BeautifulSoup(content)
print(y.mysql.host.contents[0])
for tag in y.other.preprocessing_queue:
    print(tag)

where the config.xml file might look like this:

<config>
    <mysql>
        <host>localhost</host>
        <user>root</user>
        <passwd>my secret password</passwd>
        <db>write-math</db>
    </mysql>
    <other>
        <preprocessing_queue>
            <li>preprocessing.scale_and_center</li>
            <li>preprocessing.dot_reduction</li>
            <li>preprocessing.connect_lines</li>
        </preprocessing_queue>
        <use_anonymous value="true" />
    </other>
</config>

Answer 7

Try using ReadSettings:

from readsettings import ReadSettings
data = ReadSettings("settings.json") # Load or create any json, yml, yaml or toml file
data["name"] = "value" # Set "name" to "value"
data["name"] # Returns: "value"

Answer 8

Save and load a dictionary. You will have arbitrary keys, values and arbitrary number of key, values pairs.