CODEWITHSUNDEEP
logstash
user2773013
user2773013

Reputation: 3172

can logstash process multiple output simultaneously?

i'm very new to logstash and elastic search. I am trying to store log files both in elasticsearch and a flat file. I know that logstash support both output. But are they processed simultaneously? or is it done periodically through a job?

Upvotes: 6

Views: 24104

Answers (3)

Future
Future

Reputation: 388

First you need to install output plugins:

/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch
/usr/share/logstash/bin/logstash-plugin install logstash-output-file

Then create conf files for output:

cat /etc/logstash/conf.d/nfs-output.conf

output {
  file {
    path => "/your/path/filebeat-%{+YYYY-MM-dd}.log"
  }
}


cat /etc/logstash/conf.d/30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["elasitc_ip:9200"]
    manage_template => true
    user => "elastic"
    password => "your_password"
  }
}

Then:

service logstash restart

Upvotes: 0

dobbs
dobbs

Reputation: 1043

Yes you can do this like so by tagging and cloning your inputs with the "add_tag" command on your shipper config. 

input
{
    tcp     { type => "linux" port => "50000" codec => plain { charset => "US-ASCII" } }
    tcp     { type => "apache_access" port => "50001" codec => plain { charset => "US-ASCII" } }
    tcp     { type => "apache_error"  port => "50002" codec => plain { charset => "US-ASCII" } }
    tcp     { type => "windows_security" port => "50003" codec => plain { charset => "US-ASCII" } }
    tcp     { type => "windows_application" port => "50004" codec => plain { charset => "US-ASCII" } }
    tcp     { type => "windows_system" port => "50005" codec => plain { charset => "US-ASCII" } }
udp { type => "network_equipment" port => "514" codec => plain { charset => "US-ASCII" } }
udp { type => "firewalls" port => "50006" codec => plain }
}
filter
{
    grok    { match => [ "host", "%{IPORHOST:ipaddr}(:%{NUMBER})?" ] }
    mutate  { replace => [ "fqdn", "%{ipaddr}" ] }
    dns     { reverse => [ "fqdn", "fqdn" ] action => "replace" }
    if [type] == "linux"                    { clone { clones => "linux.log" add_tag => "savetofile" } }
    if [type] == "apache_access"            { clone { clones => "apache_access.log" add_tag => "savetofile" } }
    if [type] == "apache_error"             { clone { clones => "apache_error.log" add_tag => "savetofile" } }
    if [type] == "windows_security"         { clone { clones => "windows_security.log" add_tag => "savetofile" } }
    if [type] == "windows_application"      { clone { clones => "windows_application.log" add_tag => "savetofile" } }
    if [type] == "windows_system"           { clone { clones => "windows_system.log" add_tag => "savetofile" } }
    if [type] == "network_equipment"        { clone { clones => "network_%{fqdn}.log" add_tag => "savetofile" } }
if [type] == "firewalls"        { clone { clones => "firewalls.log" add_tag => "savetofile" } }
}
output
{
    #stdout { debug => true }
    #stdout { codec => rubydebug }
    redis   { host => "1.1.1.1" data_type => "list" key => "logstash" }
}

And on your main logstash instance you would do this: 

input {
    redis {
    host => "1.1.1.1" 
    data_type => "list" 
    key => "logstash" 
    type=> "redis-input"
    # We use the 'json' codec here because we expect to read json events from redis.
    codec => json
          }
    }
    output
    {
        if "savetofile" in [tags] {
            file {
                path => [ "/logs/%{fqdn}/%{type}" ] message_format => "%{message}"   
            } 
        }
        else { elasticsearch { host => "2.2.2.2" }
    }
}

Upvotes: 6

Ban-Chuan Lim
Ban-Chuan Lim

Reputation: 7890

FYI, You can study The life of logstash event about the logstash event.

The output worker model is currently a single thread. Outputs will receive events in the order they are defined in the config file.

But the Outputs may decide to buffer events temporarily before publishing them. Ex: Output will buffers 2 or 3 events then just it write to file.

Upvotes: 0

Related Questions