CODEWITHSUNDEEP
kubernetes
user9345277
user9345277

Reputation: 73

kubernetes efk: what does fluentd's 'filter' filter?

I'm trying to enable efk in my kubernetes cluster. I find a file about fluentd's config: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml

In this file, there's:

<filter kubernetes.**>
  @id filter_kubernetes_metadata
  @type kubernetes_metadata
</filter>
# Fixes json fields in Elasticsearch
<filter kubernetes.**>
  @id filter_parser
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>

I want to use different parsers for different deployments. So I wonder:

  1. what's 'kubernetes.**' in kubernetes? Is it the name of a deployment or label of a deployment?

  2. In docker-compose file, we can tag on different containers and use the tag in fluentd's 'filter'. In kubernetes, is there any similar way?

Thanks for your help!

Upvotes: 2

Views: 235

Answers (2)

Mateo Gross
Mateo Gross

Reputation: 23

I was struggleing with the same issue as you are, but I think i found the solution for you. Kubernetes filter metadata adds a json to your log records with a lot of information, like deployment, container name and namespace name. I did like this to identify the records by namespace_name.container_name:

<filter kubernetes.**>
  @type record_modifier
  <record>
    new_tag ${record['kubernetes']['namespace_name']}_${record['kubernetes']['container_name']}
    tag ${tag}
  </record>
</filter>
<match kubernetes.**>
  @type rewrite_tag_filter
  <rule>
    key $.new_tag
    pattern /^(.+)_(.+)$/
    tag $1.$2
  </rule>
</match>

<filter *.*>
  @type record_transformer
  remove_keys new_tag
</filter>

check the documentation of record_modifier if you need more insight https://github.com/repeatedly/fluent-plugin-record-modifier

Hope this serves you :)

Upvotes: 0

mdaniel
mdaniel

Reputation: 33223

It isn't related to kubernetes, or to deployments; it is fluentd syntax that represents the top-level kubernetes "tag" and all its subkeys that are published as an event, as one can see here

Upvotes: 4

Related Questions