CODEWITHSUNDEEP
fluentdelasticsearch-opendistro
zone sd
zone sd

Reputation: 15

create index for elasticsearch as namespaces names

im useing elasticsearch opendistro whith fluentd and i want to collect my kubernetes cluster logs , i want collect logs per namespace in index's . im lookin this answer but still having problem.also i added Fluentd-${record['kubernetes']['namespace_name']} but it couldn't defined my namespaces.

im using this conf for source

    ## logs from podman
    <source>
      @type tail
      @id in_tail_container_logs
      @label @KUBERNETES
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_type string
          time_format "%Y-%m-%dT%H:%M:%S.%NZ"
          keep_time_key false
        </pattern>
        <pattern>
          format regexp
          expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
          time_format '%Y-%m-%dT%H:%M:%S.%NZ'
          keep_time_key false
        </pattern>
      </parse>
      emit_unmatched_lines true
    </source>

and about filters.conf

 <label @KUBERNETES>
      <match kubernetes.var.log.containers.fluentd**>
        @type relabel
        @label @FLUENT_LOG
      </match>


      <filter kubernetes.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
      </filter>

      <filter kubernetes.**>
        @id filter_parser
        @type parser
        key_name log
        reserve_data true
        remove_key_name_field true
        <parse>
          @type multi_format
          <pattern>
            format json
          </pattern>
          <pattern>
            format none
          </pattern>
        </parse>
      </filter>

      <match **>
        @type relabel
        @label @OUTPUT
      </match>
    </label>

and finally in output

  04_outputs.conf: |-
    <label @OUTPUT>
      <match **>
        @type elasticsearch
        host myhost
        port 9200
        user myuser
        password mypass
        scheme https
        ssl_verify false
        logstash_prefix Fluentd-${record['kubernetes']['namespace_name']}
        logstash_format true
        <buffer tag, $.kubernetes.namespace_name>
            flush_thread_count 8
            flush_interval 5s
            chunk_limit_size 2M
            queue_limit_length 32
            retry_max_interval 30
            retry_forever true
        </buffer>
      </match>
    </label>

but in index still i haven't anything

Upvotes: 0

Views: 2524

Answers (2)

Natjo
Natjo

Reputation: 2128

You are using the wrong format to specify the namespace. Instead of ${record['kubernetes']['namespace_name']} you must use ${$.kubernetes.namespace_name}. It's also documented here.

I got the correct solution from this issue thread, the output part should look something like this:

      logstash_format true
      logstash_prefix Fluentd-${$.kubernetes.namespace_name}
      <buffer tag, $.kubernetes.namespace_name>
        @type memory
        timekey 5
        timekey_wait 5
      </buffer>

I hope this helps!

Upvotes: 1

Joshua Robles
Joshua Robles

Reputation: 151

I was recently working on a fluent-bit -> fluentd -> opensearch setup so just putting my solution here.

In my case, I was also getting the literal ${record['kubernetes']['namespace_name']} as my index instead of the actual namespace (tried different variations like accessor pattern, with or without quotes, double/single etc but didn't work). If you do not need the tag, you can use it to pass the index name by rewriting it:

<match kube.**>
  @type rewrite_tag_filter
  <rule>
    key $['kubernetes']['namespace_name']
    pattern ^(.+)$
    tag $1
  </rule>
</match>

And on your output,

logstash_prefix fluentd-${tag}
logstash_format true

Hope it helps even though this can be considered a hack.

Upvotes: 3

Related Questions