CODEWITHSUNDEEP
pythonconfigurationyaml
Eduardo Pacheco
Eduardo Pacheco

Reputation: 279

How to multiply two aliases from different scopes in a YAML file

I have this YAML file as a config

training_parameters:
  epochs: 2
  learning_rate: 0.0001
dataloader_parameters:
  train_csv: ./data/metadata/metadata.csv
  val_csv: ./data/metadata/metadata.csv
  root_dir: ./data/videos
  batch_size: 4
  max_frames: &max_frames 69
  num_workers: 4
  pin_memory: True
model_parameters:
  rnn_type: lstm
  rnn_layers: 2
  rnn_hidden: &rnn_hidden 100
  trainable_backbone: false
  heads_params:
    in_features: *max_frames * *rnn_hidden
    byrt: [2, 2]
    byrn: [2, 2]
    bdrt: [2, 2]
    bdrn: [2, 2]
    ft: [2, 2]
    fn: [2, 2]
    landed: [2, 2]
    stance: [2, 2]

As you can see the in_features field inside heads_params inside model_parameters is dependent on max_frames and rnn_hidden and should be equal to their product. I tried to create the field by dereferencing and multiplying, however when reading the file in Python with:

import yaml # pip install pyyaml

with open("config.yaml", "r") as f:
    d = yaml.safe_load(f)

I got this error:

ScannerError: while scanning an alias
  in "config.yaml", line 18, column 18
expected alphabetic or numeric character, but found '*'
  in "config.yaml", line 18, column 29

In the YAML file, I already had the warning Unexpected alias at node end. How can I overcome this issue and multiply those fields directly in the YAML?

Upvotes: 0

Views: 304

Answers (1)

Anthon
Anthon

Reputation: 76882

When you specify:

*max_frames * *rnn_hidden

you are specifying an alias max_frames * *rnn_hidden, but that is not a valid alias name (and not a valid anchor name either). What you are trying to do here is what I would like to call in-scalar substitution and you cannot do that with YAML alone.

You can either specify quote the whole scalar to make it valid YAML:

in_features: '*max_frames * *rnn_hidden'

and then do so post-loading evaluation, but that will be difficult using PyYAML as the anchor information is no longer available after safe_load() finished. You could use ruamel.yaml's default round-trip mode to get that information.

Another option is specifying something like:

in_features: [*max_frames, '*', *rnn_hidden]

which will get you a Python list with resolved aliases after loading ([69, '*', 100]) which should be easy to evaluate into a single number.

Upvotes: 1

Related Questions