CODEWITHSUNDEEP
filterlogstashkibanagrok
Bhupesh Pant
Bhupesh Pant

Reputation: 4349

Log stash filters

I need a help for writing the filters for logstash. My logstash is configured to read syslog.

The log message is as follows,

Mar 14 15:11:11 localhost 192.168.235.136 {'status': 'True', 'endpoint': '/search/basic/', 'parameters': <QueryDict: {u'fileName': [u'Adware']}>, 'company': u'Global first', 'matched threat scape': [u'Enterprise IT Management and Investment'], 'request id': 11, 'user id': 2L, 'user': u' ', 'matched report id': [u'Intel-732102']}

I wanted to have filters on kibana, based on the json keys that I am passing in the message to logstash.

I am not able to write the filters to get the parameters from my logs. I have also tried http://grokdebug.herokuapp.com/ for generating the filter. It gave me the pattern that I am not sure how to use it.

{%{QS:'status'}: %{QS}, %{QS}: %{QS}, %{QS}: <QueryDict: {u%{QS}: %{SYSLOG5424SD}}>, %{QS}: u%{QS}, %{QS}: %{SYSLOG5424SD}, %{QS}: 11, %{QS}: 2L, %{QS}: u' ', 'matched report id': %{SYSLOG5424SD}}

Upvotes: 0

Views: 1385

Answers (1)

Murthy
Murthy

Reputation: 387

For the log input:

Mar 14 15:11:11 localhost 192.168.235.136 {'status': 'True', 'endpoint': '/search/basic/'}

Grok Pattern is

%{CISCOTIMESTAMP:JsonTimestamp} localhost %{IP:JsonIP} {'status': '%{WORD:JsonStatus}', 'endpoint': '%{UNIXPATH:JsonPath}'}

Please follow similar pattern approach for the rest of the fields. You can find the references at:

https://github.com/elasticsearch/logstash/blob/master/patterns/grok-patterns

Please post what fields you feel difficulty in applying grok pattern, along with following details:

i) What fields are varying and what are constants.

In the above example JsonTimestamp,JsonIP,JsonStatus and JsonPath will be indexed.

Please find below working full grok pattern for above example:

%{CISCOTIMESTAMP} localhost %{IP} {%{QS}: %{QS}, %{QS}: %{QS}, %{QS}: <QueryDict: {u%{QS}: %{SYSLOG5424SD}}>, %{QS}: u%{QS}, %{QS}: %{SYSLOG5424SD:matched_threat_scape}, %{QS}: %{NUMBER:request_id}, %{QS}: %{NUMBER:user_id}L, %{QS}: %{WORD:user}%{QS}, %{QS}: %{SYSLOG5424SD:matched_report_id}

modified changes are %{NUMBER:request_id}, %{NUMBER:user_id}L, %{WORD:user}, %{SYSLOG5424SD:matched_report_id} since 11 and 2L are not in quotes and they represent numbers, we use NUMBER to identify numeric tokens, user is represented as WORD token

Upvotes: 1

Related Questions