CODEWITHSUNDEEP
rubyelasticsearchlogstash
LuxuryWaffles
LuxuryWaffles

Reputation: 1708

Logstash split log and insert it separately into elasticsearch

I am writing a logstash config file and a log I am receiving is giving me issues, the team sent me multiple logs merged into one eg.

message: [logitem(aaa=1, bbb=1, ccc=1), logitem(aaa=2, bbb=2, ccc=2), logitem(aaa=3, bbb=3, ccc=3)]

Would it be possible to split these log into 3 and insert them individually into elasticsearch? (3 records)

Upvotes: 0

Views: 515

Answers (1)

tomr
tomr

Reputation: 563

Using a ruby filter

This way should work (see comments below for discussion and refs). You might need to tweak the grok / scan regexes in a couple of places.

grok {
    match => {
        "message" => "^\[%{GREEDYDATA:logitems}\]$"
    }
}

ruby {
    code => "event.set('logitem', event.get('message').scan(/logitem\([^\)]+\)/))"
}

split {
    field => "logitem"
}

grok {
    match => {
        "logitem" => "^logitem\(aaa=%{DATA:field_a}, bbb=%{DATA:field_b}, ccc=%{DATA:field_c}\)"
    }
}

The intention of the scan regex is to match a string that:

  • starts with logitem
  • then a ( character
  • then one or more of any character except )
  • ends with )

Using grok

This way, surprisingly, does not work. See this github issue for more detail. TL;DR... grok will not put repeated matches into an array.

filter {
  grok {
    match => {
      "message" => "^\[*(logitem\(%{DATA:logitem}\), )*logitem\(%{DATA:logitem}\)\]$"
    }
  }
  split {
    field => "logitem"
  }
}

If you are sure the messages will always have the aaa=, bbb= format you could be more explicit.

[edits 1: marked grok method as non-working and added ruby method. 2: reordered a couple of things for better flow]

Upvotes: 1

Related Questions