CODEWITHSUNDEEP
logstashkibanaelastic-stack
LeoNeo
LeoNeo

Reputation: 739

How do I group logs in Kibana/Logstash?

We have an ELK setup and the Logstash is receiving all the logs from the Filebeat installed on the server. So when I open Kibana and it asks for an index I put just a * for the index value and go to the Discover tab to check the logs and it shows each line of the log in a separate expandable section.

I want to be able to group the logs based on the timestamp first and then on a common ID that is generated in our logs per request to identify it from the rest. An example of the logs we get : 

DEBUG [2018-11-23 11:28:22,847][298b364850d8] Some information
INFO  [2018-11-23 11:27:33,152][298b364850d8] Some information
INFO  [2018-11-24 11:31:20,407][b66a88287eeb] Some information
DEBUG [2018-11-23 11:31:20,407][b66a88287eeb] Some information

I would like to see all logs for request ID : 298b364850d8 in the same drop down given they are continuous logs. Then it can break into the second dropdown again grouped by the request ID : b66a88287eeb in the order of timestamp.

Is this even possible or am I expecting too much from the tool?

OR if there is a better strategy to grouping of logs I'm more than happy to listen to suggestions.

I have been told by a friend that I could configure this in logstash to group logs based on some regex n stuff but I just don't know where and how to configure it to fo the grouping.

I am completely new to the whole ELK stack to bear with my questions which might be quite elementary in nature.

Upvotes: 1

Views: 5774

Answers (2)

WGSSAMINTHA
WGSSAMINTHA

Reputation: 190

You can use @timeStamp filter and search query as below sample image to filter what you want. enter image description here

Upvotes: -1

Gal S
Gal S

Reputation: 1040

Your question is truly a little vague and broad as you say. However, I will try to help :)

  1. Check the index that you define in the logstash output. This is the index that need to be defined Kibana - not *.
  2. Create an Index Pattern to Connect to Elasticsearch. This will parse the fields of the logs and will allow you to filter as you want.
  3. It recommend using a GUI tool (like Cerebro) to better understand what is going on in you ES. It would also help you to get better clue of the indices you have there.

Good Luck

Upvotes: 0

Related Questions