CODEWITHSUNDEEP
logstashlogstash-grok
Nick
Nick

Reputation: 189

A complicated logstash pattern in Grok

I have following 3 lines in a log that need to be grok'd for ElasticSearch through logstash.

2020-01-27 13:30:43,536 INFO  com.test.bestmatch.streamer.function.BestMatchProcessor  - Best match for ID: COi0620200110450BAD5CB723457A9B4747F1727 Total Batch Processing time: 3942
2020-01-27 13:30:43,581 INFO  HTTPConnection                                              - COi0620200110450BAD5CB723457A9B4747F1727 | People: 51 | Addresses: 5935 | HTTP Query Time: 24
2020-01-27 13:30:43,698 INFO  bestRoute                                                    - COi0620200110450BAD5CB723457A9B4747F1727 | Touch Points: 117 | Best Match Time 3943

I tried various grok patterns but couldn't get to any concrete one.

Edited as per request

I need the following in ES in the context of the specific log entry

1st line 

ID: COi0620200110450BAD5CB723457A9B4747F1727 
Total Batch Processing time: 3942

2nd Line 

ID: COi0620200110450BAD5CB723457A9B4747F1727 
People: 51 
Addresses: 5935 
HTTP Query Time: 24

3rd Line 

Touch Points 117 
Best Match Time: 3943.

The output is from a Flink log. If there are flink patterns out there then please let me know.

Upvotes: 1

Views: 295

Answers (1)

carrotcakeslayer
carrotcakeslayer

Reputation: 1008

1st line:

^%{TIMESTAMP_ISO8601:time}\s*%{LOGLEVEL:loglevel}.*ID: (?<ID>[\w\d]*).*time: (?<total_time>[\d]*)$

2nd line:

^%{TIMESTAMP_ISO8601:time}\s*%{LOGLEVEL:loglevel}.* - (?<ID>[\w]*).*People: (?<people>[\w]*).*Addresses: (?<addresses>[\d]*).*HTTP Query Time: (?<query_time>[\d]*)$

3rd line:

^%{TIMESTAMP_ISO8601:time}\s*%{LOGLEVEL:loglevel}.* - (?<ID>[\w]*).*Touch Points: (?<touch_points>[\d]*).*Best Match Time (?<best_match_time>[\d]*)$

There are many ways to parse this, this is only one approach. I would reccomend to adjust the field names I used to the new ECS. https://www.elastic.co/guide/en/ecs/current/index.html

Upvotes: 1

Related Questions