ughai
Reputation: 9880
index based on field date in elasticsearch
I am importing a syslog using logstash (version 1.5.3) into elasticsearch(version 1.7.1) using the following configuration.
input{
file {
path => "somepath\*.log"
}
}
filter{
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:message_hostname} %{DATA:message_program}(?:\[%{POSINT:message_pid}\])?: %{GREEDYDATA:user_message}" }
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
output {
elasticsearch{
cluster => somecluster
host => localhost
index => "logindex-%{+YYYY-MM-dd}"
}
}
My index is created based on the current date and time i.e. logindex-2015-08-07.
I want to create the index based on the date syslog_timestamp and not the current date using the above format {+YYYY-MM-dd}
So If the log had a timestamp 2015-01-01, my index should be created as logindex-2015-01-01 and not logindex-2015-08-07
EDIT
Log Input Used:
Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message
Logstash Debug Output
←[36mfilter received {:event=>{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]:(root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, :level=>:debug, :file=>"(eval)", :line=>"69", :method=>"filter_func"}
←[0m
←[36mRunning grok filter {:event=>#<LogStash::Event:0x166d6250 @metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root)
CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, "host"]}>>, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-1.0.0/lib/logstash/filters/grok.rb", :line=>"283", :method=>"filter"}
←[0m
←[36mRegexp match object {:names=>["SYSLOGTIMESTAMP:message_timestamp", "SYSLOGHOST:message_hostname", "DATA:message_program", "POSINT:message_pid", "GREEDYDATA:user_message"], :captures=>["Jul 27 07:49:01", "Server1", "CRON", "21009", "(root) CMD LTest Message\r"], :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=>"179", :method=>"match_and_capture"}
←[0m
←[36mfilters/LogStash::Filters::Grok: adding value to field {:field=>"received_at", :value=>["%{@timestamp}"], :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/util/decorators.rb", :line=>"28", :method=>"add_fields"}
←[0m
←[36mEvent now: {:event=>#<LogStash::Event:0x166d6250 @metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "host"], "message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009","user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message"], "message_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_timestamp"], "message_hostname"=>[{"message"=>"Jul27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_hostname"], "message_program"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTestMessage\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_program"], "message_pid"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_pid"], "user_message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "user_message"], "@timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "@timestamp"], "received_at"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01","message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "received_at"]}>>, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-1.0.0/lib/logstash/filters/grok.rb", :line=>"303", :method=>"filter"}
←[0m
←[36mDate filter: received event {:type=>nil, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter
-date-1.0.0/lib/logstash/filters/date.rb", :line=>"206", :method=>"filter"}
←[0m←[36mDate filter looking for field {:type=>nil, :field=>"syslog_timestamp", :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/
jruby/1.9/gems/logstash-filter-date-1.0.0/lib/logstash/filters/date.rb", :line=>
"209", :method=>"filter"}
←[0m
←[36moutput received {:event=>{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]:(root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, :level=>:debug, :file=>"(eval)", :line=>"76", :method=>"output_func"}
←[0m
←[36mFlushing output {:outgoing_count=>1, :time_since_last_flush=>22.048, :outgoing_events=>{nil=>[["index", {:_id=>nil, :_index=>"%index-2015-08-14", :_type=>"logs", :_routing=>nil},
#<LogStash::Event:0x166d6250
@metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={"retry_count"=>0}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @metadata={"retry_count"=>0},
@accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>sage\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r","received_at"=>"2015-08-14T07:34:53.215Z"}, "host"], "message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message"], "message_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_timestamp"], "message_hostname"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_hostname"], "message_program"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_program"], "message_pid"=>[{"message"=>"Jul 27 07:49:01 Server1CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_pid"], "user_message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "user_message"], "@timestamp"=>[{"message"=>"Jul 2707:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "@timestamp"], "received_at"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "received_at"], "type"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r","received_at"=>"2015-08-14T07:34:53.215Z"}, "type"], "syslog_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "syslog_timestamp"]}>>]]}, :batch_timeout=>1, :force=>nil, :final=>nil, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/stud-0.0.20/lib/stud/buffer.rb", :line=>"207", :method=>"buffer_flush"}
←[0m{ "message" => "Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r",
"@version" => "1",
"@timestamp" => "2015-08-14T07:34:53.215Z",
"host" => "HOST-LT",
"message_timestamp" => "Jul 27 07:49:01",
"message_hostname" => "Server1",
"message_program" => "CRON",
"message_pid" => "21009",
"user_message" => "(root) CMD LTest Message\r",
"received_at" => "2015-08-14T07:34:53.215Z"
}
Upvotes: 1
Views: 4017
Answers (1)
Val
Reputation: 217254
The problem might be that the locale on your machine is different from the locale used to produce the log. So you should specify the locale in your date filter, like this:
...
filter{
grok {
...
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
locale => "en"
}
}
...
UPDATE
Based on your log output above (very helpful!!), your date filter should work on the message_timestamp field and not syslog_timestamp (which doesn't exist)
Upvotes: 2
Related Questions
- How logstash process field for ES date_range type?
- How to check for an index with a date in logstash output
- Logstash unable to index into elasticsearch because it can't parse date
- ElasticSearch query specifying an indexname using todays date
- Index based on the value of a converted unix timestamp in logstash
- Elasticsearch monitoring indices
- elasticsearch index name with date
- Parsing a date field in logstash to elastic search
- Can I configure Logstash to use an index which includes a parameterized date?
- Logstash Date Filter