CODEWITHSUNDEEP
regexlogstashlogstash-grok
Naresh
Naresh

Reputation: 5397

Regex in config for dynamic columns in logstash

I have the log file of which i have pasted two rows below:

Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=traffic subtype=forward level=notice vd=root srcip=172.16.1.251 srcport=62032 srcintf="Combo_LAN" dstip=X.X.X.X dstport=X dstintf="wan2" sessionid=16172588 status=close user="X.X" group="Open Group" policyid=2 dstcountry="United States" srccountry="Reserved" trandisp=snat transip=X.X.X.X transport=X service=HTTP proto=6 applist="Block_Applications" duration=11 sentbyte=2377 rcvdbyte=784 sentpkt=6 rcvdpkt=7 identidx=5 utmaction=passthrough utmevent=webfilter utmsubtype=ftgd-cat urlcnt=1 hostname="tacoda.at.atwola.com" catdesc="Advertising"

Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=utm subtype=webfilter eventtype=ftgd_allow level=notice vd="root" policyid=2 identidx=5 sessionid=15536743 user="X.X" srcip=X.X.X.X srcport=X srcintf="Combo_LAN" dstip=X.X.X.X dstport=80 dstintf="wan2" service="http" hostname="streaming.sbismart.com" profiletype="Webfilter_Profile" profile="Open Group_Policy" status="passthrough" reqtype="direct" url="/diffusion/" sentbyte=984 rcvdbyte=202 msg="URL belongs to an allowed category in policy" method=domain class=0 cat=18 catdesc="Brokerage and Trading"

My question is i can parse the data if number of columns and order is fixed.

But, how do i parse the dynamic columns in the config file so that i don't get the _grokparsefailure?

Upvotes: 0

Views: 477

Answers (2)

Ban-Chuan Lim
Ban-Chuan Lim

Reputation: 7890

Ruby Plugin can help you.

Here is the configuration:

input {
    stdin{
    }
}

filter {
    ruby {
        code => '
            msg = event["message"]
            msgIndex = msg.index("date=")
            msgInsert = msg[msgIndex..-1]
            msgMap = msgInsert.scan(/(\w+)=("(.*?)"|([^ ]+))/).map { |(first, second)| [first, second] }
            for x in msgMap
                key = x[0]
                value = x[1]
                event[key] = value
            end
        '
    }
}

output {
    stdout{
        codec => rubydebug
    }
}
  1. First, get all the key=value pair by index the start value date=
  2. Then map all the key,value to string array.
  3. Use For loop to insert all the value.

I have try your logs and I can create all the correspond field with the value. Hope this can help you

Upvotes: 1

Alain Collins
Alain Collins

Reputation: 16362

The simple answer to avoiding grokparsefailure is to provide a valid pattern that matches your input. That said, your question seems to imply that the fields are not always specified in this order. Given the examples, you should be using the "kv" filter to split these key/value pairs into fields.

Upvotes: 1

Related Questions