CODEWITHSUNDEEP
bashshellloggingsedzsh
joshuatvernon
joshuatvernon

Reputation: 1621

How to pass a dynamic number of arguments to sed to use as filters

I would like to be able to easily search log files.

My log files are very large and look like so . . .

-------------------------------
id=<id>
request=<request>
...
status=<status>
metrics=<metrics>
-------------------------------
id=<id>
request=<request>
...
status=<status>
metrics=<metrics>

Each log starts with a line of dashes and then there is multiple lines of information that may change for each different log.

So I would like to be able to get output of any logs that match a dynamic number of filters.

I plan to make this a bash function I can call and pass in the filename and multiple filters

e.g. Calling it with a single filter

$ searchLogs <filename> <first-match>

e.g. Calling it with multiple filters

$ searchLogs <filename> <first-match> <second-match> <third-match> ...

I found a command that I can use to match with a single filter

$ sed '/^---/ !{H;$ !d};x;/<search_term>/ !d'

...and I could create multiple functions for different lengths and just keep appending ;/<search_term>/ !d to it for each additional search term but I'd like to build it dynamically if possible and store it in a function so I can easily share it with team members

Here is roughly what I think it would look like . . . but I'm not having any luck

function searchRequestLogs() {
    logFile="$1"
    searchTerms="${@:2}"
    sedString="'"'/^---/ !{H;$ !d};x';

    for searchTerm in $searchTerms
    do
       # append searchTerm
       sedString=$sedString';/'$searchTerm'/ !d';
    done
    sedString=$sedString"'"

    cat $logFile | sed $sedString;
}

I should also note that the log files can be very large so I'm keen not to do too many passes if possible.

Upvotes: 0

Views: 325

Answers (2)

joshuatvernon
joshuatvernon

Reputation: 1621

Here is the final solution I came up with for sed. Happy for others to find possibly better solutions. Especially, if there is any speed enhancements that can be made since some log files may be VERY large

# Search a log file with an arbitrary number of search filters
function searchRequestLog() {
    local logFile="$1"
    local filters="${@:2}"
    local sedScript='/^---/ !{H;$ !d};x';

    # loop over filters and add each to sed script
    for filter in $filters
    do
       # append filter
       sedScript=$sedScript';/'$filter'/ !d';
    done;

    # search log file for log events that contain ALL filters
    sed -e "$sedScript" "$logFile";
}

Upvotes: 0

builder-7000
builder-7000

Reputation: 7627

You could use grep rather than sed to search for text:

search_logs() {
    local logfile="$1"
    local searchterms="${@:2}"
    searchterms="${searchterms// /\\|}"
    grep "$searchterms" "$logfile"
}

Sample input:

$ cat testfile
a
b
c
d
e

Sample output:

$ search_logs testfile a d e
a
d
e

A few remarks:

  • use local to avoid polluting the global namespace
  • use parameter substitution to create the search string. For the example above is a\|d\|e

Upvotes: 1

Related Questions