How to grep/sed/awk for a range of output starting with a whitespace character

Question

I have a file that looks something like this:

# cat $file
...
ip access-list extended DOG-IN
 permit icmp 10.10.10.1 0.0.0.7 any
 permit tcp 10.11.10.1 0.0.0.7 eq www 443 10.12.10.0 0.0.0.63
 deny   ip any any log
ip access-list extended CAT-IN
 permit icmp 10.13.10.0 0.0.0.255 any
 permit ip 10.14.10.0 0.0.0.255 host 10.15.10.10
 permit tcp 10.16.10.0 0.0.0.255 host 10.17.10.10 eq smtp
...

I want to be able to search by name (using a script) to get 'section' output for independent access-lists. I want the output to look like this:

# grep -i dog $file | sed <options??>

ip access-list extended DOG-IN
 permit icmp 10.10.10.1 0.0.0.7 any
 permit tcp 10.11.10.1 0.0.0.7 eq www 443 10.12.10.0 0.0.0.63
 deny   ip any any log

...with no further output of inapplicable non-indented lines.

I have tried the following:

grep -A 10 DOG $file | sed -n '/^[[:space:]]\{1\}/p'

...Which only gives me the 10 lines after DOG which begin with a single space (including lines not applicable to the searched access-list).

sed -n '/DOG/,/^[[:space:]]\{1\}/p' $file

...Which gives me the line containing DOG, and the next line beginning with a single space. (Need all the applicable lines of the access-list...)

I want the line containing DOG, and all lines after DOG which begin with a single space, until the next un-indented line. There are too many variables in the content to depend on any patterns other than the leading space (there is not always a deny on the end, etc...).

Answer 1

A shorter, POSIX-compliant awk solution, which is a generalized and optimized translation of @Tiago's excellent Perl-based answer.

^{One advantage of these answers over the sed solutions is that they use literal substring matching rather than regular expressions, which allows passing in arbitrary search strings, without needing to worry about escaping. That said, if you did want regex matching, use the ~ operator rather than the index() function; e.g., index($0, name) would become $0 ~ name. You then have to make sure that the value passed for name either contains no accidental regex metacharacters meant to be treated as literals or is an intentionally crafted regex.}

name='DOG' # Case-sensitive name to search for.

awk -v name="$name" '/^[^[:space:]]/ {if (p) exit; if (index($0,name)) {p=1}}  p' file

• Option -v name="$name" defines awk variable name based on the value of shell variable $name (awk has no direct access to shell variables).
• Variable p is used as a flag to indicate whether the current line should be printed, i.e., whether it is part of the section of interest; as long as p is not initialized, it is treated as 0 (false) in a Boolean context.
• Pattern /^[^[:space:]]/ matches only header lines (lines that start with a non-whitespace character), and the associated action ({...}) is only processed for them:
  • if (p) exit exits processing altogether, if p is already set, because that implies that the next section has been reached. Exiting right away has the benefit of not having to process the remainder of the file.
  • if (index($0, name)) looks for the name of interest as a literal substring in the header line at hand, and, if found (in which case index() returns the 1-based position at which the substring was found, which is interpreted astruein a Boolean context), sets flagpto1({p=1}`).
• p simply prints the current line, if p is 1, and does nothing otherwise. That is, once the section header of interest has been found, it and subsequent lines are printed (up until the next section or the end of the input file).
  Note that this is an example of a pattern-only command: only a pattern (condition) is specified, without an associated action ({...}), in which case the default action is to print the current line, if the pattern evaluates to true. (That technique is used in the common shorthand 1 to simply unconditionally print the current record.)

If case-INsensitivity is needed:

name='dog' # Case-INsensitive name to search for.

awk -v name="$name" \
  '/^[^[:space:]]/ {if(p) exit; if(index(tolower($0),tolower(name))) {p=1}}  p' file

^{Caveat: The BSD-based awk that comes with macOS (still applies as of 10.12.1) is not UTF-8-aware.: the case-insensitive matching won't work with non-ASCII letters such as ü.}

GNU awk alternative, using the special IGNORECASE variable:

awk -v name="$name" -v IGNORECASE=1 \
  '/^[^[:space:]]/ {if(p) exit; if(index($0,name)) {p=1}}  p' file

---

Another POSIX-compliant awk solution:

name='dog' # Case-insensitive name of section to extract.

awk -v name="$name" '
 index(tolower($0),tolower(name)) {inBlock=1; print; next} # 1st section line found.
 inBlock && !/^[[:space:]]/       {exit}             # Exit at start of next section.
 inBlock                                             # Print 2nd, 3rd, ... section line.
 ' file

Note:

• next skips the remaining pattern-action pairs and proceeds to the next line.
• /^[[:space:]]/ matches lines that start with at least one whitespace char. As @Chrono Kitsune explains in his answer, if you wanted to match lines that start with exactly one whitespace char., use /^[[:space:]][^[:space:]]/. Also note that, despite its name, character class [:space:] matches ANY form of whitespace, not just spaces - see man isspace.
• There's no need to initialize flag variable inBlock, as it defaults to 0 in numeric/Boolean contexts.
• If you have GNU awk, you can more easily achieve case-insensitive matching by setting the IGNORECASE variable to a nonzero value (-v IGNORECASE=1) and simply using index($0, name) inside the program.

---

A GNU awk solution, IF, you can assume that all section header lines start with 'ip' (so as to break the input into sections that way, rather than looking for leading whitespace):

awk -v RS='(^|\n)ip' -F'\n' -v name="$name" -v IGNORECASE=1 '
  index($1, name) { sub(/\n$/, ""); print "ip" $0; exit }
  ' file

• -v RS='(^|\n)ip' breaks the input into records by lines that fall between line-starting instances of string 'ip'.
• -F'\n' then breaks each record into fields ($1, ...) by lines.
• index($1, name) looks for the name on the current record's first line - case-INsensitively, thanks to -v IGNORECASE=1.
• sub(/\n$/, "") removes any trailing \n, which can stem from the section of interest being the last in the input file.
• print "ip" $0 prints the matching record, comprising the entire section of interest - since, however the record doesn't include the separator, 'ip', it is prepended.

Answer 2

Using GNU sed (Linux):

name='dog'  # case-INsensitive name of section to extract
sed -n "/$name/I,/^[^[:space:]]/ { /$name/I {p;d}; /^[^[:space:]]/q; p }" file

^{To make matching case-sensitive, remove the I after the occurrences of /I above.}

• -n suppresses default output so that output must explicitly be requested inside the script with functions such as p.
• Note the use of double quotes ("...") around the sed script, so as to allow references to the shell variable $name: The double quotes ensure that the shell variable references are expanded BEFORE the script is handed to sed (sed itself has no access to shell variables).
  • Caveat: This technique is tricky, because (a) you must use shell escaping to escape shell metacharacters you want to pass through to sed, such as $ as \$, and (b) the shell-variable value must not contain sed metacharacters that could break the sed script; for generic escaping of shell-variable values for use in sed scripts, see this answer of mine, or use my awk-based answer.
• /$name/I,/^[^[:space:]]/ uses a range to match the line of interest (/$name/I; the trailing I is GNU sed's case-insensitivity matching option) through the start of the next section (/^[^[:space:]]/ - i.e., the next line that does NOT start with whitespace); since sed ranges are always inclusive, the challenge is to selectively remove the last line of the range, IF it is the start of the next section - note that this will NOT be the case if the section of interest is the LAST one in the file.
  Note that the commands inside { ... } are only executed for each line in the range.
• /$name/I {p;d}; unconditionally prints the 1st line of the range: d deletes the line (which has already been printed) and starts the next cycle (proceeds to the next input line).
• /^[^[:space:]]/q matches the last line in the range, IF it is the next section's first line, and quits processing altogether (q), without printing the line.
• p is then only reached for section-interior lines and prints them.

Note:

• The assumption is that header lines can be identified by NOT starting with a whitespace char., and that any other lines are non-header lines - if more sophisticated matching is required, see my awk-based answer.
• This solution has the slight disadvantage that the range regexes must be duplicated, although you could mitigate that with shell variables.

---

FreeBSD/macOS sed can almost do the same, except that it lacks the case-insensitivity option, I.

name='DOG'  # case-SENSITIVE name of section to extract
sed -n -e "/$name/,/^[^[:space:]]/ { /$name/ {p;d;}; /^[^[:space:]]/q; p; }" file

Note that FreeBSD/OSX sed generally has stricter syntax requirements, such as the ; after a command even when followed by }.

If you do need case-insensitivity, see my awk-based answer.

Answer 3

@mklement0 squeezed my already-inscrutable sed down to this:

sed '/^ip/!{H;$!d};x; /DOG/I!d'

which swaps accumulated multiline groups into the pattern buffer for processing -- the main logic (/DOG/I!d here) operates on whole groups.

The /^ip/! identifies continuation lines by the absence of a first-line marker and accumulates them, so the x only runs when an entire group has been accumulated.

Some corner cases don't apply here:

The first x swaps in a phantom empty group at the start. If that doesn't get dropped during ordinary processing, adding a 1d fixes that.

The last x also swaps out the last line of the file. That's usually just last line of the last group, already accumulated by the H, but if some command might produce one-line groups you need to supply a fake one at the end (with e.g. echo "header phantom" | sed '/^header/!{H;$!d};x' realdata.txt -, or { showgroups; echo header phantom; } | sed '/^header/!{H;$!d};x'.

Answer 4

I added a second answer as mklement0 pointed a flaw on my logic.

This is yet a very simple way to do that in Perl:

perl -ne ' /^\w+/ && {$p=0}; /DOG/ && {$p=1}; $p && {print}'

EXAMPLES:

cat /tmp/file  | perl -ne ' /^\w+/ && {$p=0}; /DOG/ && {$p=1}; $p && {print}'
ip access-list extended DOG-IN
 permit icmp 10.10.10.1 0.0.0.7 any
 permit tcp 10.11.10.1 0.0.0.7 eq www 443 10.12.10.0 0.0.0.63
 deny   ip any any log

cat /tmp/file  | perl -ne ' /^\w+/ && {$p=0}; /CAT/ && {$p=1}; $p && {print}'
ip access-list extended CAT-IN
 permit icmp 10.13.10.0 0.0.0.255 any
 permit ip 10.14.10.0 0.0.0.255 host 10.15.10.10
 permit tcp 10.16.10.0 0.0.0.255 host 10.17.10.10 eq smtp

EXPLANATION:

If the line starts with [a-z0-9_] set $p false

If the line contains PATTERN in this case DOG sets $p true

if $p true prints

Answer 5

The simplest way I can think of is: sed '/DOG/, /^ip/ !d' | sed '$d'

cat file | sed '/DOG/, /^ip/ !d' | sed '$d'
ip access-list extended DOG-IN
 permit icmp 10.10.10.1 0.0.0.7 any
 permit tcp 10.11.10.1 0.0.0.7 eq www 443 10.12.10.0 0.0.0.63
 deny   ip any any log

Explanation:

• first sed command prints from the line containing DOG to the next line starting with ip
• second sed command deletes the last line(which is the line starting with ip)

Answer 6

awk -vfound=0 '
/DOG/{
    found = !found;
    print;
    next
}

/^[[:space:]]/{
    if (found) {
        print;
        next
    }
}

{ found = !found }
'

You can substitute any ERE in place of /DOG/, such as /(DOG)|(CAT)/, and the rest of the script will do the work. You can condense it if you like of course.

Note that just because a line begins with a space, that doesn't mean there is only one space. /^[[:space:]]{1}/ will match the leading space, even in a string like

                      nonspace

meaning it is equivalent to /^[[:space:]]/. If your format is so rigid that there must always only be a single space, use /^[[:space:]][^[:space:]]/ instead. Lines like the one with "nonspace" above will not be matched.