CODEWITHSUNDEEP
bashvimsed
jaywalker
jaywalker

Reputation: 1146

bash - sed query to edit yaml file

I have a config.yaml file which contains among other values the following list of kafka brokers which I want to remove from the config using a bash script.

kafka.brokers:
    - "node003"
    - "node004"

I am doing this currently by invoking vi from inside the script using the command:

vi $CONF_BENCHMARK/config.yaml -c ":%s/kafka.brokers:\(\n\s*-\s".*"\)*/kafka.brokers:/g" -c ':wq!'

I understand that sed is a more appropriate tool to accomplish the same task but when I try to translate the above regex to sed, it does not work.

sed -i -e "s/kafka.brokers:\(\n\s*-\s".*"\)*/kafka.brokers:/g" $CONF_BENCHMARK/config.yaml

I am doing something wrong ?

Upvotes: 3

Views: 16316

Answers (4)

Christoffer Soop
Christoffer Soop

Reputation: 1508

Consider using yq instead of sed or awk. Deleting the key kafka.brokers then becomes as simple as:

yq d $CONF_BENCHMARK/config.yaml '"kafka.brokers"'

The following snippet demonstrates the yq delete feature:

cat <<EOF | yq d - '"kafka.brokers"'    
some:
  path: value
kafka.brokers:
  - "node003"
  - "node004"
EOF

... and results in the output

some:
  path: value

Upvotes: 1

Andreas Louv
Andreas Louv

Reputation: 47099

As other have pointed out, you will need to be explicit to get sed working with multiply lines.

The real answer is to use AWK a beautiful answer is provided by karakfa. But for the educational purpose I will provide an sed answer: 

sed  '
  /kafka.brokers/ {
    :a
    $be
    N
    /\n[[:space:]]*-[[:space:]]"[^\n]*"[^\n]*$/ba
    s/\n.*\(\n\)/\1/
    P;D
    :e
    s/\n.*//
  }
' input

Basically sed will keep append lines to the pattern space from when kafka.brokers up until \n[[:space:]]*-[[:space:]]"[^\n]*"[^\n]*$ is not matches.

This will leave the pattern space with one trailing line it in, i.e:

kafka.brokers:\n    - "node003"\n    - "node004"\nother stuff$

Replacing everything \n.*\(\n\) with a newline leaves the following pattern space:

kafka.brokers:\nother stuff$

P;D will print the first line from the pattern space and then restart the cycle with the remaning pattern space. Making the input support:

kafka.brokers:
    - "node003"
    - "node004"
kafka.brokers:
    - "node005"
more_input

Upvotes: 2

karakfa
karakfa

Reputation: 67497

awk to the rescue!

sed is line based, this should work...

$ awk 's{if(/\s*-\s*"[^"]*"/) next; else s=0} /kafka.brokers:/{s=1}1' file

Explanation

if(/\s*-\s*"[^"]*"/) next if pattern matches skip to next line
s{if(/\s... check pattern if only s is set
/kafka.brokers:/{s=1} when header seen set s
1 shorthand for print lines (if not skipped)
s{... else s=0} if s was set but pattern not found, reset s

Upvotes: 6

Ben
Ben

Reputation: 1701

Your Vim pattern matches across multiple lines, but sed works line-by-line. (That is, it first tries to match your pattern against kafka.brokers: and fails, then it tries to match - "node003", and so on.) Your instinct to use something other than Vim was right, but sed probably isn't the best tool for the job here.

This answer addresses the problem of matching multi-line patterns with sed in more detail.

My personal recommendation would be to use a scripting language like Python or Perl to deal with complicated pattern-matching. You can run a Python command with python -c <command>, for instance, just like you did with Vim, or you could write a small Python script that you call from your Bash script. It's a little more complicated than a sed one-liner, but it will probably save you a lot of debugging and make your script easier to maintain and modify.

Upvotes: 1

Related Questions