CODEWITHSUNDEEP
bashenvironment-variables
peedee
peedee

Reputation: 3739

Can envsubst not do in-place substitution?

I have a config file which contains some ENV_VARIABLE styled variables.

This is my file.
It might contain $EXAMPLES of text.

Now I want that variable replaced with a value which is saved in my actual environment variables. So I'm trying this:

export EXAMPLES=lots
envsubst < file.txt > file.txt

But it doesn't work when the input file and output file are identical. The result is an empty file of size 0.

There must be a good reason for this, some bash basics that I'm not aware of? How do I achieve what I want to do, ideally without first outputting to a different file and then replacing the original file with it?

I know that I can do it easily enough with sed, but when I discovered the envsubst command I thought that it should be perfect for my use case, so I'd like to use that.

Upvotes: 51

Views: 45297

Answers (12)

Peter Micuch
Peter Micuch

Reputation: 31

This works just fine without the need for temp file. Expression is evaluated before it flushes the content to the file.

echo "$(envsubst < file.txt)" > file.txt

Upvotes: 3

Laine Damien
Laine Damien

Reputation: 145

envsubst < file.txt | tee file.txt

Caution: @SoftwareFactor rightly pointed out in the comments that this answer suffers from a race condition and can cause the resulting file to be empty at random.

Upvotes: 4

Khalid
Khalid

Reputation: 151

I found another shortcut to put into temp file and then rename it to original file.

envsubst < in.txt > out.txt && mv out.txt in.txt

Upvotes: 15

igops
igops

Reputation: 751

To avoid creating a temporary file, use sponge not tee:

envsubst < file.txt | sponge file.txt

From https://linux.die.net/man/1/sponge:

sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.

Upvotes: 29

Alek
Alek

Reputation: 873

If you use bash, check this:

a=`<file.txt` && envsubst <<<"$a" >file.txt

Tested on 500mb file, works as expected.

Upvotes: 0

林奕忠
林奕忠

Reputation: 809

Updated 20221011 - Using 1 sed command

sed -i -r 's/["`]|\$\(/\\&/g; s/.*/echo "&"/ e' ./input.txt

Updated 20221007 - Using 2 sed commands

sed -i -r 's/["`]|\$\(/\\&/g' input.txt
sed -i -r 's/.*/echo "&"/ e' input.txt

Do it without envsubst

envsubst_file () {
    local original_file=$1
    local temp_file=$(mktemp)
    trap "rm -f ${temp_file}" 0 2 3 15
    cp -p ${original_file} ${temp_file}
    cat ${original_file} | sed -r 's/["`]|\$\(/\\&/g' | sed -r 's/.*/echo "&"/g' | sh > ${temp_file}
    mv ${temp_file} ${original_file}
}

envsubst_file 'input.txt'

First using sed to escapes double quotes("), backtick(`) and command $( by prefixing with backslash(\),then using sed again replace with

echo "&"

Finally executing the shell script and redirecting to ${temp_file}

Upvotes: 2

SoftwareFactor
SoftwareFactor

Reputation: 8588

Here is the solution that I use:

originalfile="file.txt"
tmpfile=$(mktemp)
cp --attributes-only --preserve $originalfile $tmpfile
cat $originalfile | envsubst > $tmpfile && mv $tmpfile $originalfile

Be careful with other solutions that do not use a temporary file. Pipes are asynchronous, so the file will occasionally be read after it has already been truncated.

Upvotes: 31

Frank David
Frank David

Reputation: 317

This answer was framed from two other answers. I guess this is the best solution.

originalFile=file.txt
tmpfile=$(mktemp)
cat $originalFile | envsubst > "$tmpfile" && cp -pf "$tmpfile" $originalFile
rm -f "$tmpfile"

Upvotes: 0

rici
rici

Reputation: 241701

Redirects are handled by the shell, not the program being executed, and they are set up before the program is invoked.

The redirect >output.file has the effect of creating output.file if it doesn't exist and emptying it if it does. Either way, you end up with an empty file, and that is what the program's output is redirected to.

Programs like sed which are capable of "in-place" modification must take the filename as a command-line argument, not as a redirect.

In your case, I would suggest using a temporary file and then renaming it if all goes OK.

Upvotes: 16

Matt Rundle
Matt Rundle

Reputation: 81

It's worth noting that the mv solution won't maintain file permissions. Using cp -pf would be preferable in the case that you're modifying an executable file.

tmpfile=$(mktemp)
cat file.txt | envsubst > "$tmpfile" && cp -pf "$tmpfile" file.txt
rm -f "$tmpfile"

Upvotes: 1

peedee
peedee

Reputation: 3739

In the end I found that using envsubst was too dangerous after all. My files might contain dollar signs in places where I don't want any substitution to happen, and envsubst will just replace them with empty strings if no corresponding environment variable is defined. Not cool.

Upvotes: -4

Christoph H&#246;sler
Christoph H&#246;sler

Reputation: 1274

You can achieve in-place substitution by calling envsubst from gnu sed with the "e" command:

EXAMPLES=lots sed -i 's/.*/echo & | envsubst/e' file.txt

Upvotes: 1

Related Questions