Get string from variable lists in Bash

Question

I have a file (file.env) similar to this:

kw_var1='string1'
kw_var2='string 2'
kw_var3='this is string 3'
kw_var4='does not matter'
kw_var5='maybe'

w_var1=qwert_1
w_var2=qwert_2
w_var3=qwert_3
w_var4=qwert_4

and I need to create a string list_of_values which contains the values of all variables that start with kw_, i.e.

$ echo -e $list_of_values

should output:

'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe' 

I tried to iterate over them, but cannot get this to work. My code:

list_of_values=$(for param in $(cat $file.env | grep "kw\_"); do echo $(echo -e '$'$param | cut -s -d '=' -f1); done)

but this is what I get:

 $kw_var1 $kw_var2 $kw_var3 $kw_var4 $kw_var5

Note that:

• the variable values will contain spaces;
• list_of_values will be used as an argument to another function

Any ideas with what is wrong?

UPDATE:

When doing the final echo I used:

$ echo -e $list_of_values | tr '\n' ' '

to get everything in one line.

Answer 1

Your attempt uses a few practices that aren't recommended, and contains a few syntax errors:

• You use $file.env, but it should be just file.env
• Don't use for to read lines from a file (see this article)
• echo $(cmd) is the same as just cmd plus globbing and word splitting, which often isn't what you want
• echo -e '$'$param is going to print a literal $ sign
• cut -f1 is selecting the first field, but you want the second one

This is a solution "in the spirit" of what you tried, but using just Bash:

list=$(
    while IFS='=' read -r key val; do
        [[ $key == kw_* ]] && printf '%s ' "$val"
    done < file.env
)
list=${list% }   # Remove trailing blank

If you deal with strings containing spaces, though, it's generally advisable to use an array instead. Since file.env is valid Bash, we can source the lines we're interested in and then build an array with the values of all the kw_ variables:

source <(grep '^kw_' file.env)
declare -n var
for var in "${!kw_@}"; do list+=("$var"); done

The array now contains one string per element, without the literal single quotes:

$ printf '%s\n' "${list[@]}"
string1
string 2
this is string 3
does not matter
maybe

declare -n sets var to a nameref: it is treated as if it actually were the variable whose name it holds. This requires Bash 4.3 or newer.

Answer 2

$ cat tst.awk
/^kw_/ {
    sub(/[^=]+=/,"")
    str = str sep $0
    sep = " "
}
END {
    print str
}

e.g. note that it handles this=that in your desired output string correctly:

$ cat file
kw_var1='string1'
kw_var2='string 2'
kw_var3='this is string 3'
kw_var4='does not matter'
kw_var5='maybe'
kw_var6='this=that'

w_var1=qwert_1
w_var2=qwert_2
w_var3=qwert_3
w_var4=qwert_4

$ awk -f tst.awk file
'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe' 'this=that'

---

Updated: given what you've now told us in comments, here's how I'd do it assuming you need to access individual values by their tags sometimes, otherwise you could use a regular array instead of associative:

$ cat tst.sh
#!/bin/env bash

declare -A kw
declare -A w
while IFS= read -r line; do
    tag="${line%%=*}"
    val="${line#*=}"
    case "$tag" in
        kw* ) kw["$tag"]="$val" ;;
        w*  ) w["$tag"]="$val" ;;
        ?*  ) printf 'Error: unexpected contents: "%s"\n' "$line"; exit  1;;
    esac
done < file.env

printf '\nAll kw indices => values:\n'
for idx in "${!kw[@]}"; do
    printf '\t%s => %s\n' "$idx" "${kw[$idx]}"
done

printf '\nAll kw values passed to a function (printf) at once:\n'
printf '\t%s\n' "${kw[@]}"

printf '\nAll w indices => values:\n'
for idx in "${!w[@]}"; do
    printf '\t%s => %s\n' "$idx" "${w[$idx]}"
done

printf '\nAll w values passed to a function (printf) at once:\n'
printf '\t%s\n' "${w[@]}"

.

$ ./tst.sh

All kw indices => values:
        kw_var4 => does not matter
        kw_var5 => maybe
        kw_var6 => this=that
        kw_var1 => string1
        kw_var2 => string 2
        kw_var3 => this is string 3

All kw values passed to a function (printf) at once:
        does not matter
        maybe
        this=that
        string1
        string 2
        this is string 3

All w indices => values:
        w_var3 => qwert_3
        w_var2 => qwert_2
        w_var1 => qwert_1
        w_var4 => qwert_4

All w values passed to a function (printf) at once:
        qwert_3
        qwert_2
        qwert_1
        qwert_4

The above was run on this file.env without the redundant single quotes around the values, otherwise you'd just remove them in the script:

$ cat file.env
kw_var1=string1
kw_var2=string 2
kw_var3=this is string 3
kw_var4=does not matter
kw_var5=maybe
kw_var6=this=that

w_var1=qwert_1
w_var2=qwert_2
w_var3=qwert_3
w_var4=qwert_4

---

wrt our discussion in the comments and using printf '<%s>\n' in place of fitsort which I don't know and don't have:

$ list[0]='foo bar'; list[1]='etc'

$ printf '<%s>\n' "${list[@]}"
<foo bar>
<etc>

$ printf '<%s>\n' $(printf '%s\n' "${list[@]}")
<foo>
<bar>
<etc>

$ printf '<%s>\n' "$(printf '%s\n' "${list[@]}")"
<foo bar
etc>

See how the first version correctly simply passes the contents of list[] to the fitsort-replacement command while the others pass the strings output by printf to it instead?

Answer 3

I used dynamic references.

$: out="$( . file.env; for r in ${!kw_*}; do printf "'%s' " "${!r}"; done; echo )"
$: echo "$out"

'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe'

Answer 4

Trying your code

I tried your command and get this as output :

$kw_var1
$kw_var2

$kw_var3



$kw_var4


$kw_var5

You had the wrong output because you chose the first field when you used cut instead of the second.

---

Fixing cut command

for param in $(cat test.txt | grep "kw\_"); do echo $(echo '$'$param | cut -s -d '=' -f2); done

Returns :

'string1'
'string
'this


'does

'maybe'

---

Fixing IFS

You used a for in loop but it does not iterate over newlines, it iterates over spaces. You need to change the IFS (Internal Field Separator) variable first :

IFS=$'\n'; for param in $(cat <file> | grep "kw\_"); do echo $(echo $param | cut -s -d '=' -f2); done

Output :

'string1'
'string 2'
'this is string 3'
'does not matter'
'maybe'

---

Using printf

To get the output on one line, you can use printf instead of echo :

for param in $(cat <file> | grep "kw\_"); do printf "$(echo $param | cut -s -d '=' -f2) "; done; printf "\n"

Output :

'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe' 

---

Using while

You could simplify the command and use a while read statement that iterates directly over lines :

cat <file> | grep "kw\_" | cut -d"=" -f2 | while read line; do printf "${line} "; done; printf "\n"

---

Using awk

Last but not least, you can use awk which radically simplifies your code:

awk -F"=" '/kw_/{printf "%s ", $2}END{print ""}' <file>

Output :

'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe'

If the extra space at the end of the line is annoying, you can do this :

awk -F"=" '/kw_/{printf "%s%s", delim, $2; delim=" "}END{print ""}' <file>

Awk explained :

# Using = as delimiter
awk -F"=" '
    # If line contains kw_
    /kw_/{
        # Prints second field
        printf "%s%s", delim, $2;
        delim=" "
    }
    END{
        # Prints newline
        print ""
    }' <file>

---

Final code

list_of_values=$(awk -F"=" '/kw_/{printf "%s%s", delim, $2; delim=" "}END{print ""}' $file.env)

Answer 5

Make two arrays of your bunch of variables, then you can easily iterate over them like this

#!/bin/bash

kw=(
'string1'
'string 2'
'this is string 3'
'does not matter'
'maybe'
)

w=(
'qwert_1'
'qwert_2'
'qwert_3'
'qwert_4'
)

for i in {1..5}
do
    echo -n "\"${kw[$i]}\" "
done

echo

for i in {1..4}
do
    echo -n "\"${w[$i]}\" "
done

echo