CODEWITHSUNDEEP
shellunix
gunnersfan
gunnersfan

Reputation: 61

Read and Replace key-value properties using Shell Script

I wanted to replace the keys in my template file (found recursively using find) with the corresponding values. The keys will be loaded from a properties file. See my script below:

Script file Contents:

. /home/uname/config/appl.properties
echo ${name_1}
echo ${age_1}
job_1="IT"
echo ${job_1}
for file in $(find /home/uname/config -name "*.txt"); do
  echo $file
  temp_file="/home/uname/temp_file.txt"
  cp -f $file $temp_file
  rm $file
  envsubst < $temp_file > $file
done

appl.properties Contents:

name_1="theBestName"
age_1="25"

template.txt Contents:

My name is ${name_1}, my age is ${age_1} and my job is in ${job_1}.

I am able to load the key/value pairs (the echo statement displays the key values), but when envsubst writes out the template file, it is not replacing any of the keys with the values. The output I get is:

My name is  and my age is and my job is in .

Upvotes: 0

Views: 2598

Answers (2)

Charles Duffy
Charles Duffy

Reputation: 295373

envsubst, as the name implies, requires that its key/value pairs be in environment variables. Mere assignments create shell variables, not environment variables.

The following is an attempt at a best-practices replacement:

set -a # turn on auto-export
. appl.properties
set -a # turn off auto-export

while IFS= read -r -d '' filename; do
  tempfile=$(mktemp "$filename.XXXXXX")
  if envsubst <"$filename" >"$tempfile"; then
    mv "$tempfile" "$filename"
  else
    rm -f "$tempfile"
  fi
done < <(find /home/uname/config -name '*.txt' -print0)

Key points:

  • Quotes are important. Try putting spaces in your filenames and seeing how the original behaves if you're not so sure.
  • Using set -a before sourcing the file ensures that the variables it sets are present in the environment.
  • Using IFS= read -r -d '' filename to iterate over names from a NUL-delimited source (such as find -print0) -- unlike for filename in $(find ...) -- ensures that your code correctly handles filenames with spaces, filenames with glob characters in their names, and other unusual cases.
  • Using while read ...; done < <(find ...) instead of find ... | while read avoids the bug documented in BashFAQ #24, wherein variables set in the loop do not persist. See BashFAQ #1 for more on reading streams line-by-line via this method, and/or the UsingFind page.
  • Using mktemp to generate temporary filenames prevents symlink attacks -- think of what would happen if someone else with permissions to write to /home/uname/, but without permissions to write to /etc, created a symlink to /etc/passwd (or any other file you care about) named /home/uname/temp_file.txt. Moreover, using mktemp to generate random names means that multiple concurrent instances of the same script won't stomp on the same temporary filename.
  • Writing the output of envsubst to a temporary file, and then renaming that temporary file to your destination name, ensures that you don't overwrite your output until it's finished being generated. Thus, even if the process is interrupted when partially completed, you have some level of guarantee that the output file will be left in either its original state or fully written (details dependent on your filesystem's configuration and semantics).

Upvotes: 3

Petr Skocik
Petr Skocik

Reputation: 60058

> $file truncates the file before envsubst starts. Use sponge(1) from moreutils to get around that or write your own sponge that will capture all its input before writing it out.

Upvotes: 0

Related Questions