CODEWITHSUNDEEP
linuxbashshellawksed
Decius
Decius

Reputation: 185

Using a shell script to replace tokens in a file

I have a file that holds token variables. When we switch environments we want to replace those tokens correctly for the environment we're in.

How can I do this in a Linux shell? I'm thinking awk or sed.

Example:

File has this data:

DB_HOST=__HOST__
DB_PASSWORD=__PASSWORD__

I want to read this file, recognize the __HOST__ and replace it with the environment variable $(HOST). We would get the environment variable name from the matched string between the two underscores.

Does anyone have a good idea how we could do this?

Upvotes: 7

Views: 9966

Answers (4)

john k
john k

Reputation: 6615

Here's what you want:

 while IFS= read -r line  || [[ -n "$line" ]]; do
    from=$(echo $line | cut -d '=' -f 1 )
    to=$(echo $line | cut -d '=' -f 2 )
    sedString=${sedString}" s/__${from}__/${to}/g;"
  done < tokenFile.txt

sed "${sedString}" old.txt >> new.txt

It builds a string with all the replacements you want from tokenFile.txt, then replaces them all at once with one sed command, in whatever file you specify. tokenFile.txt has your 'old=new' pairs. old.txt is the files with tokens. new.txt is where you output.

So for example, after running the script you get:

$ cat tokenFile.txt
DB_HOST=127.0.0.1

$ cat old.txt
dbconnect __DB_HOST__

$ cat new.txt
dbconnect 127.0.0.1

Upvotes: 2

JackG22
JackG22

Reputation: 11

I wrote a bash script cause I did not have envsubst available. I used grep to find my token pattern then sed to make the replacements. Note the use of bash indirect expansion.

Given file replace-vars.txt:

Craig likes ${ENV_1}
Anuj likes ${ENV_2}
Dave likes ${ENV_3}

Run the following script:

#!/usr/bin/env bash

# env variables to replace
export ENV_1=milk
export ENV_2=cheese
export ENV_3=candy

# File to modify
declare FILE_TO_REPLACE=replace-vars.txt

# Pattern for your tokens -- e.g. ${token}
declare TOKEN_PATTERN='(?<=\$\{)\w+(?=\})'

# Find all tokens to replace
declare TOKENS=$(grep -oP ${TOKEN_PATTERN} ${FILE_TO_REPLACE} | sort -u)

# Loop over tokens and use sed to replace
for token in $TOKENS
do
  echo "Replacing \${${token}} with ${!token}"
  sed -i "s/\${${token}}/${!token}/" ${FILE_TO_REPLACE}
done

Output:

Replacing ${ENV_1} with milk
Replacing ${ENV_2} with cheese
Replacing ${ENV_3} with candy

Modified replace-vars.txt:

Craig likes milk
Anuj likes cheese
Dave likes candy

Upvotes: 1

rici
rici

Reputation: 241861

The program envsubst was designed for roughly this purpose, but it wants to to use standard shell format for the strings to be substituted from the environment:

# Here-doc delimiter quoted to avoid having the shell do the substitutions
$ HOST=myhost PASSWORD=secret envsubst <<"EOF"
DB_HOST=$HOST
DB_PASSWORD=${PASSWORD}
EOF

DB_HOST=myhost
DB_PASSWORD=secret

envsubst is part of Gnu gettext, and is widely available. If you really need to use the __...__ syntax, it would be easy to change it to envsubst syntax by piping through sed:

$ cat myfile.txt
DB_HOST=__HOST__
DB_PASSWORD=__PASSWORD__

$ sed -E 's/__(([^_]|_[^_])*)__/${\1}/g' myfile.txt |envsubst

Alternatively, if you have Python, there is a very simple solution:

from sys import stdin
from os import getenv
import re

pattern = re.compile(r"__(\w*)__")

def replacer(m):
    val = getenv(m[1])
    if val is None:
        # No such environment
        # Perhaps produce a warning, or an error
        # Here we just pass through the string unaltered
        return m[0]
    return val

print(pattern.sub(replacer, stdin.read()))

Example:

$ export HOST=myhost PASSWORD=secret 
$ python3 subst.py < myfile.txt
DB_HOST=myhost
DB_PASSWORD=secret

The key to this solution is that Python (and only a few other languages, unfortunately) allows the replacement argument of pattern substitution to be a function, which is then called on each successive match to produce the replacement text. That makes writing functions like this easier in Python than in awk, for example, where the replacement text is fixed.

Upvotes: 12

that other guy
that other guy

Reputation: 123550

This is what m4 is meant for:

$ export HOST=myhost PASSWORD=mypassword

$ cat myfile
DB_HOST=__HOST__
DB_PASSWORD=__PASSWORD__

$ m4 -D __HOST__="$HOST" -D __PASSWORD__="$PASSWORD" myfile
DB_HOST=myhost
DB_PASSWORD=mypassword

If you want to autodefine macros for all environment variables, that's of course possible if slightly iffy:

$ cat mymacros.m4
esyscmd(`env | sed -ne "s/^\([A-Z_]*\)=\(.*\)/define(__\1__, \`\2')/p" | tr -d "\n"')

$ cat myfile2
include(`mymacros.m4')
DB_HOST=__HOST__
DB_PASSWORD=__PASSWORD__
And other arbitrary text containing things like "My editor is __EDITOR__"

$ m4 myfile2
DB_HOST=myhost
DB_PASSWORD=mypassword
And other arbitrary text containing things like "My editor is vim"

It's not necessarily the simplest way of doing it, but it's highly flexible and extensible, with support for conditional statements, macro functions and more.

Upvotes: 5

Related Questions