CODEWITHSUNDEEP
fb-hydra
Jana
Jana

Reputation: 11

hydra - structured config group/package override via yaml

I'm not very successfully trying to figure out how to achieve an override of a group/package with and via a yaml file. Trying to explain my problem using the example (files and folder structre) from the hydra documentation https://hydra.cc/docs/tutorials/structured_config/schema/.

yaml.config as:

defaults:
  - base_config                 # --> reference to dataclass
  - db: base_mysql              # --> reference to dataclass
  - _self_ debug: true

gives the expected (print when running the myapp.py):

db:
  driver: mysql
  host: localhost
  port: 3306
  user: ???
  password: ???

Using the yaml file instead instead of the base_mysql dataclass is also fine thus the yaml.config as:

defaults:
  - base_config
  - db: mysql                   # --> reads db/mysql.yaml
  - _self_
debug: true

prints again as expected

db:
  driver: mysql
  host: localhost
  port: 3306
  user: omry
  password: secret

Overriding individual fields is as well fine, e.g. with config.yaml like

defaults:
  - base_config
  - db: mysql
  - _self_
debug: true
db:
  password: UpdatedPassword

What I'm to able to figure out is how to override the full db group with a/via another yaml file - defining the structure via a dataclass and then override/set the values like:

defaults:
  - base_config
  - db: base_mysql      # --> reference to dataclass to define the structure
  - _self_
debug: true
db: mysql        # --> mysql.yaml

throws the following error: In 'config': Validation error while composing config:

Merge error: str is not a subclass of MySQLConfig. value: mysql
    full_key: 
    object_type=Config

Searching the internet/stackoverflow already showed me that moving the self to the first position will get rid of the error - but then the composition order is "wrong".

Keeping the order as it is and using the mysql.yaml for an override works well - when done via commandline (python myapp.py db=mysql when the line "db:mysql" is not present), but for my usecase it much more convinient to handle it all via the yaml file(s).

Somehow I assume that the same functionality is available via CLI and files/code and that I just did not mange to figure out how it works.

(hydra version 1.1 in a conda environment with python 3.9)

Thank you very much in advance for any help that you can provide.

Upvotes: 1

Views: 2481

Answers (1)

Jasha
Jasha

Reputation: 7649

If understand correctly, you want to use the defaults list in your primary yaml file to merge together the base_mysql config with the mysql config. This will do the trick:

defaults:
  - base_config
  - db: [base_mysql, mysql]
  - _self_
debug: true

Passing a list [base_mysql, mysql] of config names causes those configs base_mysql and mysql to be merged together. This is documented here -- see the "CONFIG_NAMES" alternative for specifying an option in the defaults list.

Note that passing the CLI override db=mysql (as in python myapp.py db=mysql) results in modification of the defaults list; the resulting defaults list will be the same as if you had used the following in your yaml file:

defaults:
  - base_config
  - db: mysql
  - _self_
debug: true

You can pass a list [base_mysql, mysql] of config names at the CLI like this:

python my_app.py 'db=[base_mysql, mysql]'

Upvotes: 0

Related Questions