CODEWITHSUNDEEP
fluentd
Terence Bor
Terence Bor

Reputation: 96

Fluentd log source

I'm using Fluentd for shipping two types of logs to Elasticsearch cluster (application and other logs).

Logs located in the same folder /var/log/containers/ and have same name format e.g: app-randomtext.log, dts-randomtext.log etc .

I'd like to assign different indices to them to separate app logs from any other that present now or will appear in this folder.

Here is my try to make a wildcard for "path" in the block, but it doesn't work. Could anybody point me where is my mistake? Thanks 

##source for app logs
  <source>
  @type tail
  path /var/log/containers/app*.log
  pos_file /var/log/fluentd-containers-app.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag app.*
  keep_time_key true
  format json
</source>

##source for everything else
<source>
  @type tail
  path /var/log/containers/!(app*.log)
  pos_file /var/log/fluentd-containers-non-app.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag non-app.*
  keep_time_key true
  format json
</source>

<match app.**>
  @type "aws-elasticsearch-service"
  type_name "kube-fluentd-aws-es"
  index_name app
  include_tag_key true
  tag_key "@log_name"
  @log_level info
  <endpoint>
    url "#{ENV['ELASTICSEARCH_URL']}"
    region "#{ENV['ELASTICSEARCH_REGION']}"
    access_key_id "#{ENV['ELASTICSEARCH_ACCESS_KEY']}"
    secret_access_key "#{ENV['ELASTICSEARCH_SECRET_KEY']}"
  </endpoint>
</match>

<match non-app.**>
  @type "aws-elasticsearch-service"
  type_name "kube-fluentd-aws-es"
  index_name non-app
  include_tag_key true
  tag_key "@log_name"
  @log_level info
  <endpoint>
    url "#{ENV['ELASTICSEARCH_URL']}"
    region "#{ENV['ELASTICSEARCH_REGION']}"
    access_key_id "#{ENV['ELASTICSEARCH_ACCESS_KEY']}"
    secret_access_key "#{ENV['ELASTICSEARCH_SECRET_KEY']}"
  </endpoint>
</match>

I expect Fluentd to follow tail of all the logs in the folder, but with this config Fluentd follows tail only for app-randomtext.log

Thanks

Upvotes: 3

Views: 887

Answers (1)

Terence Bor
Terence Bor

Reputation: 96

Finally I managed to find an answer. exclude_path is what I need.

    ##source for everything else
    <source>
      @type tail
      path /var/log/containers/*.log
      exclude_path ["/var/log/containers/app*.log"]
      pos_file /var/log/fluentd-containers-non-app.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      tag non-app.*
      keep_time_key true
      format json
    </source>

Here Fluentd follows all the *.log files excluding those that starts with app

Upvotes: 2

Related Questions