CODEWITHSUNDEEP
arraysfiltersplitlogstash
Tw K
Tw K

Reputation: 70

How to split the custom logs and add custom field name in each values on logstash

I want to split the custom logs

"2016-05-11 02:38:00.617,userTestId,Key-string-test113321,UID-123,10079,0,30096,128,3"

that log means

Timestamp, String userId, String setlkey, String uniqueId, long providerId, String itemCode1, String itemCode2, String itemCode3, String serviceType

I try to made a filter using ruby

filter {
        ruby{
        code => "
        fieldArray = event['message'].split(',')
        for field in fieldArray
                result = field
                event[field[0]] = result
end
"
}
}

but I don't have idea how to split the logs with adding field names each custom values as belows.

Timestamp : 2016-05-11 02:38:00.617
userId : userTestId
setlkey : Key-string-test113321
uniqueId : UID-123
providerId : 10079
itemCode1 : 0
itemCode2 : 30096
itemCode3 : 128
serviceType : 3

How can I do?

Thanks regards.

Upvotes: 0

Views: 843

Answers (1)

baudsp
baudsp

Reputation: 4100

You can use the grok filter instead. The grok filter parse the line with a regex and you can associate each group to a field.
It is possible to parse your log with this pattern : 

grok {
    match => {
        "message" => [
           "%{TIMESTAMP_ISO8601:timestamp},%{USERNAME:userId},%{USERNAME:setlkey},%{USERNAME:uniqueId},%{NUMBER:providerId},%{NUMBER:itemCode1},%{NUMBER:itemCode2},%{NUMBER:itemCode3},%{NUMBER:serviceType}"
        ]
    }
}

This will create the fields you wish to have.
Reference: grok patterns on github
To test : Grok constructor

Another solution :
You can use the csv filter, which is even more closer to your needs (but I went with grok filter first since I have more experience with it): Csv filter documentation

The CSV filter takes an event field containing CSV data, parses it, and stores it as individual fields (can optionally specify the names). This filter can also parse data with any separator, not just commas.

I have never used it, but it should look like this :

csv {
   columns => [ "Timestamp", "userId", "setlkey", "uniqueId", "providerId", "itemCode1", "itemCode2 "itemCode3", "serviceType"  ]
}

By default, the filter is on the message field, with the "," separator, so there no need to configure them.

I think that the csv filter solution is better.

Upvotes: 1

Related Questions