CODEWITHSUNDEEP
logstashelastic-stack
Junior Cloud Enginner
Junior Cloud Enginner

Reputation: 3

Filter Logstash logs based on minimum log level (not specific levels)

I'm currently using ELK to monitor my cluster and experiencing an overwhelming volume of logs. As a way to streamline the data, I'd like to filter and retain only logs with a severity level of "warn" or above. I'm specifically looking for a method to achieve this filtering within Logstash without the need to explicitly list each individual log level.

I managed to drop logs with a specific log level but I don't want to specify the exact levels, only the min level.

filter {
  if [log.level] == "info" {
    drop { }
  }
}

Is there a solution that can dynamically handle different minimum severity thresholds?

Thanks in advance!

Upvotes: 0

Views: 183

Answers (1)

Badger
Badger

Reputation: 4072

If you can create a list of all possible log levels and their abbreviations then yes, it can be done.

    ruby {
        init => '
            Levels = [
                /^emer/,
                /^alert/,
                /^critical/,
                /^fatal/,
                /^error/,
                /^warn/,
                /^notice/,
                /^info/,
                /^debug/,
                /^trace/
            ]
            Threshold = Levels.find_index(/^warn/)
        '
        code => '
            level = event.get("[log][level]")
            if level
                level = level.downcase
                levelNum = -1
                Levels.each_index { |x| if Levels[x].match? (level); levelNum = x; end }
                if levelNum > Threshold
                    event.cancel
                end
            end
        '
    }

Note that the default levelNum of -1 is less than the Threshold, so unrecognized log levels will not be dropped.

Upvotes: 0

Related Questions