CODEWITHSUNDEEP
bashsed
bob.sacamento
bob.sacamento

Reputation: 6661

sed substitution: substitute string is a variable needing expansion AND contains slashes

I am fighting with sed to do a substitution where the substitute string contains slashes. This general topic has been discussed on stack overflow before. But, AFAICT, I have anew wrinkle that hasn't been addressed in previous questions.

Let's say I have a file, ENVIRO.tpml, which has several lines, one of which is

Loaded modules: SUPPLY_MODULES_HERE

I want to replace SUPPLY_MODULES_HERE in an automated fashion with a list of loaded modules. (At this point, if anyone has a better way to do this than sed, please let me know!) My first effort here is to define an environment variable and use sed to put it into the file:

> modules=$(module list 2>&1)
> sed "s/SUPPLY_MODULES_HERE/${modules}/" ENVIRO.tmpl > ENVIRO.txt

(The 2>&1 being needed because module list sends its output to STDERR, for reasons I can't begin to understand.) However, as is often the case, the modules have slashes in them. For example

> echo ${modules}
gcc/9.2.0  mpt/2.20

The slashes kill my command because sed can't understand the expression and thinks my substitution command is "unterminated".

So I do the usual thing and use some other character for the command delimiter:

> modules=$(module list 2>&1)
> sed "s|SUPPLY_MODULES_HERE|${modules}|" ENVIRO.tmpl > ENVIRO.txt

and I still get an "unterminated 's'" error.

So I replace double quotes with single quotes:

> sed 's|SUPPLY_MODULES_HERE|${modules}|' ENVIRO.tmpl > ENVIRO.txt

and now I get no error, but the line in ENVIRO.txt looks like

Loaded modules: ${modules}

Not what I was hoping for.

So, AFAICT, I need double quotes to expand the variable, but I need single quotes to make the alternative delimiters work. But I need both at the same time. How do I get this?

UPDATE: Gordon Davisson's comment below got to the root of the matter: "echo ${modules} can be highly misleading". Examining $modules with declare -p shows that it actually has a newline (or, more generally, some kind of line break) in it. What I did was add an extra step to extract newlines out of the variable. With that change, everything worked fine. An alternative would be to convince sed to expand the variable with line breaks and substitute it as such into the text, but I haven't been able to make that work. Any takers?

Upvotes: 1

Views: 99

Answers (1)

anubhava
anubhava

Reputation: 786091

sed is not the best tool here due to use of regex and delimiters.

Better to use awk command that doesn't require any regular expression.

awk -v kw='SUPPLY_MODULES_HERE' -v repl="$(module list 2>&1)" '
n = index($0, kw) {
   $0 = substr($0, 1, n-1) repl substr($0, n+length(kw))
} 1
' file
  • index function uses plain string search in awk.
  • substr function is used to get substring before and after the search keyword.

Upvotes: 4

Related Questions