CODEWITHSUNDEEP
logstashelastic-stack
user2518618
user2518618

Reputation: 1407

Automatic detection of types of logs in logstash

I am new to logstash, elasticsearch and kibana (ELK).

I know that I can create filters that parse specific logs and extract some fields from them. It looks like for each type of log I have to configure a specific filter. As I have around 20 different services, each writing around a hundred of different types of log this looks too difficult to me.

For type of logs I mean logs that have a specific template with parameters that change

This is a example of some logs:

Log1: User Peter has logged in

Log2: User John has logged in

Log3: Message "hello" sent by Peter

Log4: Message "bye" sent by John

I want ELK to discover automatically that here we have two types of log

Type1: User %1 has logged in

Type2: Message "%1" sent by %2

Is that possible? Is there any example to do that? I don't want to write manually the template for each type of log, I want it to be discovered automatically.

Then also extract the parameters. This is what I wold like to see in the index

Log1: Type1, params: Peter

Log2: Type1, params: John

Log3: Type2, params: hello, Peter

Log4: Type2, params: bye, John

After that I would like ELK to scan again my index and discover that param %1 of Type1 is usually param %2 in Type2 (the user name). Also it should discover that Log1 and Log3 are related (same user).

The last thing it should do is finding unusual sequences of actions (logins without the corresponding logout, for example)

Is any of this possible without having to manually configure all types of logs? If not, can you point me to some example of this multipass indexing even if it involves manual configuration?

Upvotes: 0

Views: 575

Answers (1)

sysadmin1138
sysadmin1138

Reputation: 1303

Logstash has no discovery like this, you'll have to do the language parsing yourself. It's tedious and repetitive, but it gets the job done. You have a few options here, depending on your ability to influence other areas:

  • If the format of those logs is changeable, consider pushing for an authentication-logging standard. That way you only need one pattern.
  • Consider a modular approach to generating your filter pipeline. Log1 patterns go in one module, Log2 in another. It makes maintainability easier.

You have my sympathy with this problem. I've had to integrate Logstash with the authentication-logging of many systems by now, and each one describes what they're doing somewhat differently, all based on the whim of the developer who wrote it (which may have happened 25 years ago in some cases).

For the products we develop, I can at least influence how the logging looks. Moving away from a natural language grok format to something else, such as kv or even json goes a long way towards simplifying the parsing problem or me. The trick is convicing people that we only look at the logs through Kibana anyway, why do we need:

User %{user} logged into application %{app} in zone %{zone}

When we can have

user="%{user}" app="%{app}" zone=%{zone}

Or even:

{ "user": %{user}, "app": %{app}, "zone": %{zone} }

Since that's what it'll be when Logstash is done with it anyway.

Upvotes: 1

Related Questions