Sed variable expansion and \n match when trying to append some text

Question

I've the following template file

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#



# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
#http_access allow localnet
#http_access allow localhost

I want to append some text after the line bellow using sed (this is a requirement, I can't use another tool)

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

I'm tried to use the following command:

sed "/# INSERT YOUR OWN RULE/aTEXTTOAPPEND" test.txt

The result:

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
TEXTTOAPPEND
#

The old code used perl to do the job

perl -i -0pe 's/(#( |\t)*\n# INSERT YOUR OWN RULE.*\n#( |\t)*\n)/\1\n'"$REPLACE"'\n/g' /etc/squid/squid.conf

I'm facing two problems:

1 - Not being able to use a varible in this append command

I tried so far:

$ echo $REPLACE
TEXTTOAPPEND

$ sed "/# INSERT YOUR OWN RULE/a${REPLACE}" test.txt
sed: -e expression #1, char 53: unknown command: `B'

sed "/# INSERT YOUR OWN RULE/a\${REPLACE}" test.txt
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
${REPLACE}
#

2 - Not being able to match the hash character after the line I want to append the text (when I add the \n the command stop matching, so, nothing is added in this case)

sed "/# INSERT YOUR OWN RULE.*\n#/aTEXTTOAPPEND" test.txt
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

Expected output:

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
TEXTTOAPPEND

Can someone point what I'm missing here? For variable expansion I thought using double quotes and ${var} would do the job.

---

Update:

I was trying everything using git-bash, when trying in a real linux machine the command bellow worked:

echo $REPLACE
ABC

sed "/# INSERT YOUR OWN RULE.*/a${REPLACE}" test.txt
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
ABC
#

the only problem now is the 2. how to work with the \n in this case?

sed "/# INSERT YOUR OWN RULE.*\n#.*/a${REPLACE}" test.txt
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

Answer 1

sed "/INSERT YOUR OWN RULE/a '\n'TEXTTOAPPEND" file-for-change |
sed "/INSERT YOUR OWN RULE/{n;s/'/#/;n;s/^'//;n;d"}

Second sed is for change and removing "'" leading symbol at start of next two lines

Or, more simple variant:

sed -r '/INSERT YOUR/,+1{s/(#$)/\1\nTEXTTOAPPEND/}'

It works with GNU sed.

Answer 2

Sometimes the simplest way is the best, maybe this is the case?

sed '8aTEXTTOAPPEND' -i test.txt

This will add 'TEXTTOAPPEND' after 8-th string.

And automate this

N=$(grep -n 'INSERT YOUR OWN RULE' test.txt) # get line number
N=${N%%:*}                                   # remove all except line number
((N++))                                      # inc line number coz we got this line with #
sed "${N}aTEXTTOAPPEND" -i test.txt          # add text after $Nth line

Oneliner

N=$(grep -n 'INSERT YOUR OWN RULE' test.txt); N=${N%%:*}; ((N++)); sed "${N}aTEXTTOAPPEND" -i test.txt

Answer 3

This might work for you (GNU sed):

cat <<! | sed 'N;/^# INSERT YOUR OWN RULE.*\n#/!{P;D};r /dev/stdin' file
Text to be appended or a variable
$var
or both $var
!

Construct a here document to be appended and pipe it through to the sed invocation. The sed invocation uses the N and the P;D commands to open a two line window throughout the length of the file but on matching the required two lines invokes the r command which appends the former here document.

An alternative:

sed '1{x;s/^/'"$var"'/;x};N;/^# INSERT YOUR OWN RULE.*\n#/!{P;D};G' file

But,this should work too:

sed 'N;/^# INSERT YOUR OWN RULE.*\n#/!{P;D};a\'"$var" file

Answer 4

Can someone point what I'm missing here?

Commands in sed are separated by a newline. Sed sees a newline - it assumes the command ends here. I can reproduce your sed: -e expression #1, char 53: unknown command: 'B' with simple variable that has a newline and the next line starts with B:

replace="something
B something"
sed "/# INSERT YOUR OWN RULE/a${replace}"

sed sees asomething and appends something to the output and newline terminates the a command. Then it sees B something and tries to parse that as a command, but B is invalid.

The most safest way to append content of the variable with sed is to use a temporary file with r command. Note, that you need a newline after r command after filename, because any ; will be parsed as part of the filename! In bash, you can be smart and combine process substitution with a here string to create a temporary file descriptor*. To output current pattern space and read next line into pattern space in sed use n command. Like this:

sed '/# INSERT YOUR OWN RULE/{n;r'<(cat <<<"$replace")$'\n}'

It will append the content of replace after the next line after the regex. Note that after r there is $'\n' - a newline.

* Just a <(echo "$replace") would work too, but I somehow feel the cat <<<"$replace" will be better in memory consumption for big strings, I didn't check that in any way.

The following script:

replace="anything
can
be here!"

cat <<EOF |
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
EOF
sed '/# INSERT YOUR OWN RULE/{n;r'<(cat <<<"$replace")$'\n}'

outputs on repl:

#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
anything
can
be here!

# Example rule allowing access from your local networks.

how to work with the \n in this case?

Sed reads one line at a time. As this is very similar, I would just point to this answer I did just yesterday that deals with the same problem. The script in sed this case would need to buffer two lines at a time with N command:

sed '
  : restart
  N # buffer two lines
  : loop
      # match two lines
      /# INSERT YOUR OWN RULE.*\n#/{
           r'<(cat <<<"$replace")'
           # print and start over
           n ; b restart
       }
       # hold, print leading line, change, remove leading line
       h ; s/\n.*// ; p ; x ; s/[^\n]*\n//
       # append next line and loop
       N
  b loop
'

But you can't do rsomething or asomething with sed -z because then records would be separated by zero, so sed reads the whole file, so asomething would be displayed after the whole file. Well, you can test it sed -z '/# INSERT YOUR OWN RULE.*\n#.*/r'<(cat <<<$replace) and it will just print the content of $replace on the end of the file.