CODEWITHSUNDEEP
fb-hydra
Natalia Levine
Natalia Levine

Reputation: 3

specializing configuration with files instead of variables in Hydra config

I'd like to use specialized configuration as per Hydra documentation in Common Patterns -> Specializing Configuration. The difference is that my specialized configuration is in a file, not just one variable. In the example below I want to choose transform based on the model and the dataset. The configs for different transforms are in files. This would work if I specified all the transform configuration in dataset_model/cifar10_alexnet.yaml file, but that would defeat the purpose because I can't reuse the transform config in this case. Alsewhere in Hydra if you specify the name of the file it would automatically pick up the config in that file, but it does not seem to work in the specialized configuration.

I've modified the example in documentation as follows:

config.yaml:

defaults:
  - dataset: cifar10
  - model: alexnet
  - transform: crop
  - dataset_model: ${defaults.0.dataset}_${defaults.1.model}
    optional: true

Added directory called transform and two files inside that directory:

crop.yaml:

# @package _group_
type: crop
test1: 7

resize.yaml:

# @package _group_
type: resize
test1: 50

changed file dataset_model/cifar10_alexnet.yaml:

# @package _global_
model:
  num_layers: 5
transform: resize

Everything else is exactly as per the documentation. When I run this I get an exception:

Traceback (most recent call last):
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/hydra/_internal/config_loader_impl.py", line 720, in _merge_config
    ret = OmegaConf.merge(cfg, loaded_cfg)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 321, in merge
    target.merge_with(*others[1:])
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 331, in merge_with
    self._format_and_raise(key=None, value=None, cause=e)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 329, in merge_with
    self._merge_with(*others)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 347, in _merge_with
    BaseContainer._map_merge(self, other)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 296, in _map_merge
    dest.__setitem__(key, src_value)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 262, in __setitem__
    self._format_and_raise(key=key, value=value, cause=e)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 694, in format_and_raise
    _raise(ex, cause)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ValidationError: 
    full_key: transform
    reference_type=Optional[Dict[Union[str, Enum], Any]]
    object_type=dict

So, the question is - is this functionality supported and if it is, what am i doing wrong?

Upvotes: 0

Views: 1756

Answers (1)

Omry Yadan
Omry Yadan

Reputation: 33646

Your config is trying to merge the string "resize" into a dictionary like:

transform:
  type: crop
  test1: 7

This is not something you can do.

You are not explaining what you are trying to do very well, but my guess is that you want to compose a different transform based on selected dataset.

Hydra 1.1 will add support for recursive defaults list which will probably allow you to do what you want.

This is the doc for the new defaults list. You can install this version as a pre-release (see primary project readme).

Upvotes: 0

Related Questions