CODEWITHSUNDEEP
externallogstashkibana
Anders Johansson
Anders Johansson

Reputation: 3

Using external lookup in Logstash

I'm currently working on two logstash projects, one is monitoring IIS logs and another one a firewall.

Now the IIS logs are from high-usage servers generating about 25GB of logs each month and there are several of these. The issue here is that we do not want to enable reverse lookup, not on the servers nor in Logstash, but from an external service as we can cache outside of the DNS lookup´function in logstash.

The other problem we want to solve with the firewall project related to the lookup of standard, and non-standard ports. Our firewall just generates a dest portnumber that we would like to translate to make our Kibana dashboards more readable. The firewall has around 10Gb/s traffic and generates a lot of syslog traffic.

We currently run 8-16 workers on our logstash server. Is there a easy (?) way to make an API call from logstash and is that even worth considering based on performance?

Another option I'm condering is "offline" batch processing, ie running batch jobs directly towards elasticsearch, but that would most likley mean I should have a separate instance of elasticsearch or redis before FrontEnd.

The best option however would most likley be to do the translation in the Kibana interface, as a scripted field, but as of what I understand that would not work for my usercases?

Upvotes: 0

Views: 704

Answers (1)

Alain Collins
Alain Collins

Reputation: 16362

The dns{} filter uses the local machine's resolution, so you couldn't integrate it with your non-DNS cache without making a new filter or dropping to ruby{}.

Depending on the number of values that you have, you could publish them to a file and use translate{}, but I would only recommend that for something like a private network lookup.

If your DNS data is in elasticsearch, you can query it during your filtering and add fields to your events that way.

For your firewall port problem, you didn't give an example of the original and desired values, but again, check out translate{} or drop to ruby{}.

Upvotes: 0

Related Questions