logstash kv filter, converting strings to integers using dynamic mapping

Question

I have a log with a format similar to:

name=johnny amount=30 uuid=2039248934

The problem is I am using this parser on multiple log files with each basically containing numerous kv pairs.

Is there a way to recognize when values are integers and cast them as such without having to use mutate on every single key value pair?(Rather than a string)

I found this link but it was very vague in where the template json file was suppose to go and how I was to go about using it. Can kv be told to auto-detect numeric values and emit them as numeric JSON values?

Answer 1

You can use ruby plugin to do it.

input {
    stdin {}
}

filter {
    ruby {
        code => "
            fieldArray = event['message'].split(' ');
            for field in fieldArray
                name = field.split('=')[0];
                value = field.split('=')[1];
                if value =~ /\A\d+\Z/
                    event[name] = value.to_i
                else
                    event[name] = value
                end
            end
        "
    }
}
output {
    stdout { codec => rubydebug }
}

First, split the message to an array by SPACE. Then, for each k,v mapping, check whether the value is numberic, if YES, convert it to Integer.

Here is the sample output for your input:

{
       "message" => "name=johnny amount=30 uuid=2039248934",
      "@version" => "1",
    "@timestamp" => "2015-06-25T08:24:39.755Z",
          "host" => "BEN_LIM",
          "name" => "johnny",
        "amount" => 30,
          "uuid" => 2039248934
}

Update Solution for Logstash 5:

input {
    stdin {}
}

filter {
    ruby {
        code => "
            fieldArray = event['message'].split(' ');
            for field in fieldArray
                name = field.split('=')[0];
                value = field.split('=')[1];
                if value =~ /\A\d+\Z/
                    event.set(name, value.to_i)
                else
                    event.set(name, value)
                end
            end
        "
    }
}
output {
    stdout { codec => rubydebug }
}

Answer 2

Note, if you decide to upgrade to Logstash 5, there are some breaking changes:

https://www.elastic.co/guide/en/logstash/5.0/breaking-changes.html

In particular, it is the event that needs to be modified to use either event.get or event.set. Here is what I used to get it working (based on Ben Lim's example):

input {
    stdin {}
}

filter {
    ruby {
        code => "
            fieldArray = event.get('message').split(' ');
            for field in fieldArray
                name = field.split('=')[0];
                value = field.split('=')[1];
                if value =~ /\A\d+\Z/
                    event.set(name, value.to_i)
                else
                    event.set(name, value)
                end
            end
        "
    }
}
output {
    stdout { codec => rubydebug }
}