CODEWITHSUNDEEP
rubyyaml
phil swenson
phil swenson

Reputation: 8894

Accessing environment variables in a YAML file for Ruby project (using ${ENVVAR} syntax)

I am building an open source project using Ruby for testing HTTP services: https://github.com/Comcast/http-blackbox-test-tool

I want to be able to reference environment variables in my test-plan.yaml file. I could use ERB, however I don't want to support embedding any random Ruby code and ERB syntax is odd for non-rubyists, I just want to access environment variables using the commonly used Unix style ${ENV_VAR} syntax.

e.g.

order-lunch-app-health:
  request:
    url: ${ORDER_APP_URL}
    headers: 
      content-type: 'application/text'
    method: get
  expectedResponse:
    statusCode: 200
    maxRetryCount: 5

All examples I have found for Ruby use ERB. Does anyone have a suggestion on the best way to deal with this? I an open to using another tool to preprocess the YAML and then send that to the Ruby application.

Upvotes: 4

Views: 8026

Answers (2)

DannyB
DannyB

Reputation: 14776

I believe something like this should work under most circumstances:

require 'yaml'

def load_yaml(file)
  content = File.read file
  content.gsub! /\${([^}]+)}/ do
    ENV[$1]
  end

  YAML.load content
end

p load_yaml 'sample.yml'

As opposed to my original answer, this is both simpler and handles undefined ENV variables well.

Try with this YAML:

# sample.yml
path: ${PATH}
home: ${HOME}
error: ${NO_SUCH_VAR}

Original Answer (left here for reference)

There are several ways to do it. If you want to allow your users to use the ${VAR} syntax, then perhaps one way would be to first convert these variables to Ruby string substitution format %{VAR} and then evaluate all environment variables together.

Here is a rough proof of concept:

require 'yaml'

# Transform environments to a hash of { symbol: value }
env_hash = ENV.to_h.transform_keys(&:to_sym)

# Load the file and convert ${ANYTHING} to %{ANYTHING}
content = File.read 'sample.yml'
content.gsub! /\${([^}]+)}/, "%{\\1}"

# Use Ruby string substitution to replace %{VARS}
content %= env_hash

# Done
yaml = YAML.load content
p yaml

Use it with this sample.yml for instance:

# sample.yml
path: ${PATH}
home: ${HOME}

There are many ways this can be improved upon of course.

Upvotes: 2

Anthon
Anthon

Reputation: 76578

Preprocessing is easy, and I recommend you use a YAML loaderd/dumper based solution, as the replacement might require quotes around the replacement scalar. (E.g. you substitute the string true, if that were not quoted, the resulting YAML would be read as a boolean).

Assuming your "source" is in input.yaml and your env. variable ORDER_APP_URL set to https://some.site/and/url. And the following script in expand.py:

import sys
import os
from pathlib import Path
import ruamel.yaml

def substenv(d, env):
    if isinstance(d, dict):
        for k, v in d.items():
            if isinstance(v, str) and '${' in v:
                d[k] = v.replace('${', '{').format(**env)
            else:
                substenv(v, env)
    elif isinstance(d, list):
        for idx, item in enumerate(d):
            if isinstance(v, str) and '${' in v:
                d[idx] = item.replace('${', '{').format(**env)
            else:
                substenv(item, env)


yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(Path(sys.argv[1]))

substenv(data, os.environ)
yaml.dump(data, Path(sys.argv[2]))

You can then do:

python expand.py input.yaml output.yaml

which writes output.yaml:

order-lunch-app-health:
  request:
    url: https://some.site/and/url
    headers:
      content-type: 'application/text'
    method: get
  expectedResponse:
    statusCode: 200
    maxRetryCount: 5

Please note that the spurious quotes around 'application/text' are preserved, as would be any comments in the original file.

Quotes around the substituted URL are not necessary, but the would have been added if they were.

The substenv routine recursively traverses the loaded data, and substitutes even if the substitution is in mid-scalar, and if there are more than substitution in one scalar. You can "tighten" the test:

        if isinstance(v, str) and '${' in v:

if that would match too many strings loaded from YAML.

Upvotes: 0

Related Questions