CODEWITHSUNDEEP
concatenationyaml
Geo
Geo

Reputation: 96817

How can I do string concatenation, or string replacement in YAML?

I have this:

user_dir: /home/user
user_pics: /home/user/pics

How could I use the user_dir for user_pics? If I have to specify other properties like this, it would not be very DRY.

Upvotes: 78

Views: 192736

Answers (11)

Janek Paluch
Janek Paluch

Reputation: 1

If you need this specifically for paths in linux, you can use also (PyYaml in used in this case):

import yaml

# Definition:

class YamlPath:
  yamlTag = '!path'
  nodes = {}

  def __init__(self, loader:yaml.Loader, node:yaml.Node):
    nodeVal = loader.construct_scalar(node)
    if "=" in nodeVal:
      key, value = nodeVal.split("=", 2)
      self.path = value.format(**self.nodes)
      self.nodes[key] = self.path
    else:
      self.path = nodeVal.format(**self.nodes)

  def __repr__(self):
    return self.path

  # To make it usable directly with os.path module
  def __fspath__(self):
    return self.path

yaml.add_constructor(YamlPath.yamlTag, YamlPath)

# Usage:

testYaml = yaml.load("""
defs:
- !path "userHome=/home/user"
- !path "userDir={userHome}/testDir"
testNode:
  testVal: !path "{userDir}"
  testList:
  - !path "{userDir}/a.xyz"
  - !path "{userDir}/z.abc"
""", yaml.Loader)
print(str(testYaml))

Result:

{'defs': [/home/user, /home/user/testDir], 'testNode': {'testVal': /home/user/testDir, 'testList': [/home/user/testDir/a.xyz, /home/user/testDir/z.abc]}}

With this example you shall be able to prepare also more complex use cases.

Upvotes: 0

user2502636
user2502636

Reputation: 115

If you are using python with PyYaml, joining strings is possible within the YAML file. Unfortunately this is only a Python solution, not a universal one:

with os.path.join:

user_dir: &home /home/user
user_pics: !!python/object/apply:os.path.join [*home, pics]

with string.join (for completeness sake - this method has the flexibility to be used for multiple forms of string joining:

user_dir: &home /home/user
user_pics: !!python/object/apply:string.join [[*home, pics], /]

Upvotes: 9

Sandesh-Gowda0094
Sandesh-Gowda0094

Reputation: 76

yaml files do support variable substitution but it is executed in a lazy approach by default.

the syntax for variable substitution in yaml file is

# this is test.yaml file and its contents.
server:
  host: localhost
  port: 80

client:
  url: http://${server.host}:${server.port}/
  server_port: ${server.port}
  # relative interpolation
  description: Client of ${.url}

if we use this default lazy approach:

from omegaconf import OmegaConf

conf = Omegaconf.load("test.yaml")

print(f"type: {type(conf).__name__}, value: {repr(conf)}")
print(f"url: {conf.client.url}\n")
print(f"server_port: {conf.client.server_port}\n")
print(f"description: {conf.client.description}\n")

output:

type: DictConfig, value: {'server': {'host': 'localhost', 'port': 80}, 'client': {'url': 'http://${server.host}:${server.port}/', 'server_port': '${server.port}', 'description': 'Client of ${.url}'}}

url: http://localhost:80/
server_port: 80
description: Client of http://localhost:80/

Notice now when we accessed and printed the values the variable as been substituted.

but when we want to pass the entire dict for a parameter then we should use this approach:

from omegaconf import OmegaConf

conf = Omegaconf.load("test.yaml")
conf = OmegaConf.to_container(conf, resolve=True)
print(conf)

# output
type: dict, value: {'server': {'host': 'localhost', 'port': 80}, 'client': {'url': 'http://localhost:80/', 'server_port': 80, 'description': 'Client of http://localhost:80/'}}

Upvotes: 0

link89
link89

Reputation: 1763

Here is the example of join tag implementation in Python with ruamel.yaml

from ruamel.yaml import YAML

class JoinTag:
    """a tag to join strings in a list"""

    yaml_tag = u'!join'

    @classmethod
    def from_yaml(cls, constructor, node):
        seq = constructor.construct_sequence(node)
        return ''.join([str(i) for i in seq])

    @classmethod
    def to_yaml(cls, dumper, data):
        # do nothing
        return dumper.represent_sequence(cls.yaml_tag, data)

    @classmethod
    def register(cls, yaml: YAML):
        yaml.register_class(cls)


if __name__ == '__main__':
    import io
    f = io.StringIO('''\
base_dir: &base_dir /this/is/a/very/very/long/path/
data_file: !join [*base_dir, data.csv]
    ''')
    yaml = YAML(typ='safe')
    JoinTag.register(yaml)
    print(yaml.load(f))

And the output will be

{'base_dir': '/this/is/a/very/very/long/path/', 'data_file': '/this/is/a/very/very/long/path/data.csv'}

Upvotes: 1

Chris Johnson
Chris Johnson

Reputation: 21956

It's surprising, since the purpose of YAML anchors & references is to factor duplication out of YAML data files, that there isn't a built-in way to concatenate strings using references. Your use case of building up a path name from parts is a good example. There must be many such uses.

Fortunately there's a simple way to add string concatenation to YAML via user-defined tags.

User-defined tags is a standard YAML capability - the YAML 1.2 spec says YAML schemas allow the "use of arbitrary explicit tags". Handlers for those custom tags need to be implemented in a custom way in each language you're targeting. Doing that in Python looks like this:

## in your python code

import yaml

## define custom tag handler
def join(loader, node):
    seq = loader.construct_sequence(node)
    return ''.join([str(i) for i in seq])

## register the tag handler
yaml.add_constructor('!join', join)

## using your sample data
yaml.load("""
user_dir: &DIR /home/user
user_pics: !join [*DIR, /pics]
""")

Which results in:

{'user_dir': '/home/user', 'user_pics': '/home/user/pics'}

You can add more items to the array, like " " or "-", if the strings should be delimited.

Something similar can be done in other languages, depending on what their parsers can do.

There are several comments along the lines of "this seems wrong since YAML is a standard, implementation-neutral language". Actually that's not what YAML is. YAML is a framework for mapping YAML schemas (consisting of tags) to implementation-specific data types, e.g. how does int map to of Python, Javascript, C++ etc. There are multiple standard YAML schemas, and which one(s) are supported by a parser is an implementation decision. When it's useful, you can create schemas with additional custom tags, and of course that requires additional parser implementation. Whether adding custom tags is a good idea or not depends on your use case. The capability exists in YAML; whether and how to apply it is up to you. Use good judgement :).

Upvotes: 52

codermarcos
codermarcos

Reputation: 287

A solution similar to @Chris but using Node.JS:

const yourYaml = `
user_dir: &user_home /home/user
user_pics: !join [*user_home, '/pics']
`;

const JoinYamlType = new jsyaml.Type('!join', {
    kind: 'sequence',
    construct: (data) => data.join(''),    
})

const schema = jsyaml.DEFAULT_SCHEMA.extend([JoinYamlType]);

console.log(jsyaml.load(yourYaml, { schema }));
<script src="https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.1.0/js-yaml.min.js"></script>

To use yaml in Javascript / NodeJS we can use js-yaml:

import jsyaml from 'js-yaml';
// or
const jsyaml = require('js-yaml');

Upvotes: 0

bmiselis
bmiselis

Reputation: 412

As of August 2019:

To make Chris' solution work, you actually need to add Loader=yaml.Loader to yaml.load(). Eventually, the code would look like this:

import yaml

## define custom tag handler
def join(loader, node):
    seq = loader.construct_sequence(node)
    return ''.join([str(i) for i in seq])

## register the tag handler
yaml.add_constructor('!join', join)

## using your sample data
yaml.load("""
user_dir: &DIR /home/user
user_pics: !join [*DIR, /pics]
""", Loader=yaml.Loader)

See this GitHub issue for further discussion.

Upvotes: 3

bryan
bryan

Reputation: 130

string.join() won't work in Python3, but you can define a !join like this:

import functools
import yaml

class StringConcatinator(yaml.YAMLObject):
    yaml_loader = yaml.SafeLoader
    yaml_tag = '!join'
    @classmethod
    def from_yaml(cls, loader, node):
        return functools.reduce(lambda a, b: a.value + b.value, node.value)

c=yaml.safe_load('''
user_dir: &user_dir /home/user
user_pics: !join [*user_dir, /pics]''')
print(c)

Upvotes: 3

Adam
Adam

Reputation: 81

I would use an array, then join the string together with the current OS Separator Symbol

like this:

default: &default_path "you should not use paths in config"
pictures:
  - *default_path
  - pics

Upvotes: 7

Brian Wells
Brian Wells

Reputation: 830

You can use a repeated node, like this:

user_dir: &user_home /home/user
user_pics: *user_home

I don't think you can concatenate though, so this wouldn't work:

user_dir: &user_home /home/user
user_pics: *user_home/pics

Upvotes: 69

Arnis Lapsa
Arnis Lapsa

Reputation: 47587

Seems to me that YAML itself does not define way to do this.

Good news are that YAML consumer might be able to understand variables.
What will use Your YAML?

Upvotes: 5

Related Questions