CODEWITHSUNDEEP
elasticsearchlogstashlogstash-groklogstash-configurationfilebeat
Moshe Saada
Moshe Saada

Reputation: 39

Logstash filter - half json line parse

I'm using 'filebeat' as a shipper an the client send it to redis, read from redis with logstash and send it to ES.

I'm trying to parse the following example line:

09:24:01.969 watchdog - INFO - 100.140.2 PASSED: Mobile:Mobile[].popover["mc1814"].select(2,) :706<<<<<<<<<<<<<<<<<<< {"actionDuration":613}

In the end I want to have a field names: "actionDuration" with the value: 613.

As you can see it's partially json. - I've tried to use grok filter, with add_field and match and I've tried to change a few configurations in the filebeat and logstash.

I'm using the basic configurations: filebeat.conf:

filebeat.prospectors:

  • input_type: log

    paths:

    • /sketch/workspace/sanity-dev-kennel/out/*.log

    fields:

    • type: watchdog
    • BUILD_ID: 82161

If there's a possibility to do it in the filebeat side I prefer, but it's also good in the Logstash side.

Thanks a lot, Moshe

Upvotes: 1

Views: 848

Answers (1)

sysadmin1138
sysadmin1138

Reputation: 1303

This sort of partial-formatting is best handled on the Logstash side, not the shipper. The filters/transforms available in FileBeat aren't up to that. A Logstash filter pipeline is, though.

filter {
  grok {
    match => {
      "message" => [ "(?<plain_prefix>^.*?) (?<json_segment>{.*$)"]
    }
  }
  json {
    source => "json_segment"
  }
  mutate {
    remove_field => [ "json_segment" ]
  }
}

This basic example will split your incoming message into two fields. a plain_prefix and a json_segment. The json{} filter is then used to parse the JSON data into the event. Finally, a mutate {} filter is used to remove the json_segment field from the event, as it has already been parsed and included.

Note: the .*? in the plain_prefix is critical in this filter. Constructed this way, everything from the first { onward is considered part of the JSON segment. If you use .*, the JSON segment will be everything from the last {, which will be a problem with complex JSON datastructures.

Upvotes: 0

Related Questions