CODEWITHSUNDEEP
elasticsearchlogstash-grokfilebeatgrok
rocky
rocky

Reputation: 163

Extract timestamp from log message

I am trying to index log files to Elastic search. All the log entries are being indexed into a field named message. @timestamp field shows the time the entry was indexed and not the timestamp from log entry.

I created a ingest pipeline with grok processor to define the pattern of the log entry. I have tried several patterns and am unable to get this working, particularly because i am new to grok.

Log sample

2019-08-05 00:04:06 info [index.js]: Request: HTTP GET /
2019-08-05 00:04:06 error [error.js]: No authorization token was found

Ingest pipeline with grok & date processor

"description" : "Extracting date from log line"
, "processors": [
{
"grok": {
"field": "message",
"patterns": ["%{yyyy-mm-dd HH:mm:ss:logtime} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"]
},
"date": {
"field": "logtime",
"target_field": "@timestamp",
"formats": ["yyyy-mm-dd HH:mm:ss"]
}
}
]
}

All i want is the ability to extract the timestamp from the log message and everything else can be ignored or wildcarded or stored in just one variable like message. So essentially indexing the log file should index the timestamp from the log message and rest of the message can stay as text or string in one field, no need to parse rest of the message.

Any help would be appreciated.

Upvotes: 2

Views: 5820

Answers (2)

rocky
rocky

Reputation: 163

I made this below change and the log messages are getting indexed now. Although i do not understand how, appreciate if someone can shed some light on it

I had the pipeline: "pipelinename" setting in Elasticsearch output section of the filebeat config file. I moved that line to filebeat inputs section right under file path section, like so

filebeat.inputs: -type: log paths: - D:\home\site\wwwroot\logs*.log pipeline: "redate"

And the log messages are getting indexed now.

Upvotes: 1

LinPy
LinPy

Reputation: 18578

use this as grok pattern:

%{TIMESTAMP_ISO8601:logtime} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}

use thes to set the timestamps :

date{
      match => ["logtime", "yyyy-MM-dd HH:mm:ss", "ISO8601"]
      timezone => "Europe/Berlin"
      target => "@timestamp"
    }

you may change the timezone to yours

Upvotes: 2

Related Questions