CODEWITHSUNDEEP
logstashelastic-stacklogstash-configuration
Leo
Leo

Reputation: 930

Logs are ignoring input section in config files

I have a simple setup for capturing logs though HTTP and TCP. I've created 2 conf files at /etc/logstash/conf.d/ (see below) but logs sent though HTTP are also being passed through the TCP pipeline and vise versa. For example when I send a log through TCP it ends up both in http-logger-* index and in tcp-logger-*.. it makes no sense to me :(

http_logger.conf

input {
  http {
    port => 9884
  }
}
filter {
    grok {
      match => ["[headers][request_path]", "\/(?<component>[\w-]*)(?:\/)?(?<env>[\w-]*)(?:\/)?"]
    }
}
output {
    amazon_es {
        hosts => ['XXXXX']
        region => 'us-west-2'
        aws_access_key_id => 'XXXXX'
        aws_secret_access_key => 'XXXXX'
        index => 'http-logger-%{+YYYY.MM.dd}'
    }
    stdout { codec => rubydebug }
}

tcp_logger.conf

input {
  tcp {
    port => 9885
    codec => json
  }
}
filter {

}
output {
    amazon_es {
        hosts => ['XXXXX']
        region => 'us-west-2'
        aws_access_key_id => 'XXXXX'
        aws_secret_access_key => 'XXXXX'
        index => 'tcp-logger-%{+YYYY.MM.dd}'
    }
    stdout { codec => rubydebug }
}

Any ideas on what am I missing? Thank you

Upvotes: 0

Views: 222

Answers (2)

Leo
Leo

Reputation: 930

The explanation provided by @Ram is spot on however there is a cleaner way of solving the issue: enter pipelines.yml.

By default it looks like this:

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

basically it loads and combines all *.conf files - in my case I had two.

To solve the issue just separate the pipelines like so: 

- pipeline.id: httplogger
  path.config: "/etc/logstash/conf.d/http_logger.conf"
- pipeline.id: tcplogger
  path.config: "/etc/logstash/conf.d/tcp_logger.conf"

The pipelines are now running separately :)

P.S. Don't forget to reload logstash after any changes here

Upvotes: 0

Ram
Ram

Reputation: 1115

The Input, filter and Output configuration even when split across a different file the logstash while processing it will process it as a single big configuration as if all the input, filter and output is specified in a single file.

So said that the event coming into logstash will pass through all the output and filter plugin configured, in your case, each event picked up by the TCP and HTTP input plugin will pass through filter plugin and output plugin configured in both http_logger.conf and tcp_logger.conf, that's the reason you are seeing events stashed in both http-logger-* and tcp-logger-* index

So in order to fix this, we can specify a unique type field for events picked by both tcp and http input plugins and then apply the filter and output plugin selectively using the type set in the input plugin as shown below

http_logger.conf

input {
  http {
    port => 9884
    type => "http_log"
  }
}

filter {
    if [type] == "http_log"
    {
       grok {
         match => ["[headers][request_path]", "\/(?<component>[\w-]*)(?:\/)?(?<env>[\w-]*)(?:\/)?"]
       }
    }
}

output {
    if ([type] == "http_log")
    {
       amazon_es {
           hosts => ['XXXXX']
           region => 'us-west-2'
           aws_access_key_id => 'XXXXX'
           aws_secret_access_key => 'XXXXX'
           index => 'http-logger-%{+YYYY.MM.dd}'
       }
    }
    stdout { codec => rubydebug }
 }

tcp_logger.conf

input {
  tcp {
    port => 9885
    codec => json
    type => "tcp_log"
  }
}

output {
  if ([type] == "tcp_log")
  {
    amazon_es {
        hosts => ['XXXXX']
        region => 'us-west-2'
        aws_access_key_id => 'XXXXX'
        aws_secret_access_key => 'XXXXX'
        index => 'tcp-logger-%{+YYYY.MM.dd}'
    }
  }
    stdout { codec => rubydebug }
}

Upvotes: 1

Related Questions