CODEWITHSUNDEEP
bashshellloopsyaml
user3868051
user3868051

Reputation: 1249

iterate through yaml using shell

I have a config_file.yml file as:

sample:
    sql: "select * from dbname.tableName where sampleDate>='2018-07-20';"
    config: {'hosts': [!!python/tuple ['192.162.0.10', 3001]]}

sample2:
    sql: "select * from dbname.tableName where sampleDate<='2016-05-25';"
    config: {'hosts': [!!python/tuple ['190.160.0.10', 3002]]}

I want to iterate through its key value pairs till EOF using shell script. Basically I want to be able to iterate through each sql till EOF, and execute each sql in a shell loop.

Tried looking through a lot of docs but they dnt have enough info how to loop through yaml using shell.

Any ideas or example will be very helpful...

Thanks!

EDIT:

I am already using ->

parse_yaml() {
local prefix=$2
local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
sed -ne "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p" $1 |
awk -F$fs '{
  indent = length($1)/2;
  vname[indent] = $2;
  for (i in vname) {
     if (i > indent) {delete vname[i]}}
  if (length($3) > 0) {
     vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
     printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
  }
}'
}
# read yaml file
eval $(parse_yaml config_file.yml "")
# access yaml content
echo $sample_sql

I am not able to understand how I can iterate through this.

Upvotes: 1

Views: 13483

Answers (3)

SwiftNinjaPro
SwiftNinjaPro

Reputation: 851

I modified the script a bit to include the keys in a space separated list.

note: this method requires a prefix.

function parse_yaml {
  local prefix=$2
  local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
  sed -ne "s|^\($s\):|\1|" \
    -e "s|^\($s\)\($w\)$s:$s[\"']\(.*\)[\"']$s\$|\1$fs\2$fs\3|p" \
    -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p"  $1 |
  awk -F$fs '{
    indent = length($1)/2;
    vname[indent] = $2;
    for (i in vname) {if (i > indent) {delete vname[i]}}
    if (length($3) > 0) {
      vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
      printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
      printf("%1$s%2$s=\"${%1$s%2$s} %3$s\"\n", "'$prefix'",vn, $2, $3);
      printf("%1$s=\"${%1$s} %2$s\"\n", "'$prefix'",vn, $2, $3);
    }
  }'
}

usage

eval $(parse_yaml "config_file.yml" "CONF_")

# to specify all keys (apart from the final level)
for key in $CONF_; do
  echo "$key"
done

# or to list keys of a specific object
for key in $CONF_sample_; do
  echo "$key"
done

Note: This method is limited to the core level, and final level. Someone else may be able to modify this to use in between levels if they need that feature.

Upvotes: 0

moebius
moebius

Reputation: 2269

Any reason why you can't use python?

It should come pre-installed with most linux distributions, and it means you don't have to re-invent the wheel!

To setup:

pip install pyyaml

Then your script is:

import yaml
f = open('config_file.yml')
yaml_file = yaml.safe_load(f)
for sample in yaml_file:
    print yaml_file[sample]["sql"]

To run:

python <script_name>.py

Upvotes: 5

nbari
nbari

Reputation: 26965

You could use grep or pcregrep with something like this:

$ while read -r line; do
  echo $line
done < <(grep -oP "sql: \"\K.+?(?=\")" config_file.yml)

Output: 

select * from dbname.tableName where sampleDate>='2018-07-20';
select * from dbname.tableName where sampleDate<='2016-05-25';

In macOS, you could use pcregrep -o "sql: \"\K.+?(?=\")" config_file.yml

The \K can be read as excluding everything to the left before it and return only the right part .+?(?=\") until " is found.

Now if your yaml is more complex you could give a try too yq (pip install yq) and then, for example, to get the sql values you could do:

$ yq -r '.sample.sql' config_file.yml

Upvotes: 0

Related Questions