CODEWITHSUNDEEP
elasticsearchkubernetesfilebeat
Djent
Djent

Reputation: 3477

Filebeat on kubernetes - excluding namespaces doesn't work

I have a filebeat outside of the kubernetes cluster, installed as an application on the host. I want to ignore two namespaces in filebeat, since they are very large and I don't need them within elastichsearch.

Here is my input definition in filebeat.yml: 

- type: log
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  json.message_key: log
  json.keys_under_root: true
  processors:
    - add_kubernetes_metadata:
        in_cluster: false
        host: main-backend
        kube_config: /etc/kubernetes/admin.conf
    - drop_event.when.regexp:
        or:
          - kubernetes.namespace: "kube-system"
          - kubernetes.namespace: "monitoring"

However, I still see a lot of log from those namespaces within my elasticsearch. Is there any way to debug it why is it happening?

Upvotes: 1

Views: 3143

Answers (2)

Claudio Kuenzler
Claudio Kuenzler

Reputation: 902

I'm using this snippet in filebeat.yml and it works. Note that I didn't add the kubernetes_metadata, just the docker_metadata (which contains kubernetes labels, too).

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.
#
processors:
  - add_docker_metadata: ~
#  - add_kubernetes_metadata: ~
  - drop_event:
        when:
          equals:
            container.labels.io_kubernetes_pod_namespace: "kube-system"
          or:
            equals:
              container.labels.io_kubernetes_pod_namespace: "monitoring"

In this case logs coming from containers within the "cattle-prometheus" namespace will be dropped.

Upvotes: 0

P Ekambaram
P Ekambaram

Reputation: 17689

Can you try as given below

- drop_event:
          when:
            or:
            - not:
                equals:
                  kubernetes.namespace: "kube-system"
            - not:
                equals:
                  kubernetes.namespace: "monitoring"
            - regexp:
                kubernetes.pod.name: "filebeat-*"
            - regexp:
                kubernetes.pod.name: "elasticsearch-*"

Upvotes: 2

Related Questions