CODEWITHSUNDEEP
logstashlogstash-configurationlogstash-file
Philipp
Philipp

Reputation: 21

logstash generic output filename

I'd like to collect the logs of multiple servers on one logstash node. As output I'd like to store one file per server. In the logs, I got a "source_host" field that indicates which server has produced the log.

As output, I'd like to get a bunch of files named by "source_host". The source hosts change frequently, so I'd need a generic configuration

e.g. logs originating from server "foo" should be saved in /logs/foo and logs from server "bar" in /logs/bar

I tried the config like this, but the file gets named "%{source_host}". When using %{host}, the file gets the hostname of the collecting server.

output{
  file {
    path => "/tmp/%{source_host}"
  }
}

Upvotes: 0

Views: 1171

Answers (1)

fylie
fylie

Reputation: 1715

My configuration:

input {
  tcp {
    port => 5544
    codec => json_lines
  }
}

output{
  file {
    path => "/tmp/%{source_host}"
  }
}

outputs to a file /tmp/foo with your sample log. 

echo '{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z", "source_host":"foo","message":"testmsg"}' | nc localhost 5544

EDIT: Here are my test results:

pancake$ echo '{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z", "source_host":"foo","message":"testmsg"}' | nc localhost 5544
pancake$ cat /tmp/foo
{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z","source_host":"foo","message":"testmsg","port":56716}
pancake$ echo '{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z", "source_host":"bar","message":"testmsg"}' | nc localhost 5544
pancake$ echo '{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z", "source_host":"bar","message":"one more message!"}' | nc localhost 5544
pancake$ cat /tmp/bar
{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z","source_host":"bar","message":"testmsg","port":56717}
{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z","source_host":"bar","message":"one more message!","port":56718}

EDIT 2: Ooh, I just thought of something. You said earlier that you aren't using any filters, right? You need to use a filter of some kind or another, otherwise the field source_host won't exist. If you have codec => json_lines (because your logs are JSON) in your input block, as I do in my example, it will parse your JSON into key value pairs. If you have no filter or codec, the entire body of the log will be stored in the message field, unmodified. Try adding an input codec and see if that helps.

Upvotes: 1

Related Questions