CODEWITHSUNDEEP
kubernetesfluent-bitkubernetes-namespace
Oren
Oren

Reputation: 113

How to exclude namespace from fluent-bit logging

Is there a way to exclude certain namespaces in fluent-bit? I would like to exclude certain namespaces, so that fluent-bit doesn't forward all logs created in those namespaces to ELK.

Is there a way to do it besides adding annotation to each pod in that namespace? I'm aware that you can update all of the pods annotations in a namespace via kubectl.

kubectl annotate pods --namespace=pks-system --all fluentbit.io/exclude='true'

Upvotes: 11

Views: 30789

Answers (5)

apr_1985
apr_1985

Reputation: 1962

Old question I know. But this works for me and may help others.

[FILTER]
    Name           kubernetes
    Match          kube.*
    # We need the full DNS suffix as Windows only supports resolving names with this suffix
    # See: https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#dns-limitations
    Kube_URL       https://kubernetes.default.svc.cluster.local:443
[FILTER]
    Name           grep
    Match          kube.*
    Exclude        $kubernetes['namespace_name'] kube-system

Upvotes: 3

iamabhishek
iamabhishek

Reputation: 447

I think the following input plugin configuration can do this:

 [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Exclude_Path      /var/log/containers/*_<myappnamespace>_*.log,/var/log/containers/*_<myappnamespace2>_*.log
        Tag               kube.infra.<namespace_name>.<pod_name>.<container_name>
        Tag_Regex         (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
        Parser            cri
        DB                /var/log/flb_kube_infra.db
        Mem_Buf_Limit     500KB
        Skip_Long_Lines   On
        Refresh_Interval  10

Found it here: https://github.com/fluent/fluent-bit/issues/758

The Exclude_Path property defines the name of the namespace for which logs will be ignored. For multiple logs use comma separated

Upvotes: 12

LoganMzz
LoganMzz

Reputation: 1619

You have achieve namespace exclusion with a combination of the three filters kubernetes, nest and grep

[FILTER]
Name                kubernetes
Match               kube.*
Kube_URL            https://kubernetes.default.svc:443
Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix     kube.var.log.containers.
Merge_Log           Off
Merge_Log_Key       log_processed
K8S-Logging.Parser  On
K8S-Logging.Exclude On

[FILTER]
Name                nest
Match               *
Wildcard            pod_name
Operation lift
Nested_under kubernetes
Add_prefix   kubernetes_

[FILTER]
Name                grep
Match               kube.*
Exclude             kubernetes_namespace_name kube-system

Upvotes: 5

Zarathon Maia
Zarathon Maia

Reputation: 31

You must read this: https://docs.fluentbit.io/manual/filter/kubernetes#kubernetes-annotations At documentation: "Request to Fluent Bit to exclude or not the logs generated by the Pod. This option will only be processed if Fluent Bit configuration (Kubernetes Filter) have enabled the option K8S-Logging.Exclude."

Upvotes: 2

mario
mario

Reputation: 11138

According to official Fluent Bit documentation, for the moment it is actually the unique way of requesting that the log processor skips the logs from certain Pods. I searched through it and found nothing but this fragment.

In addition to that, there is even a feature request raised on their GitHub project so for now we can hope it will be available in a future release.

In documentation there is only example of a separate Pod definition but for sure you should be able to apply it to Pod template in Deployment definition so you don't have to apply it to each Pod separately or to every Pod in certain namespace using the kubectl command you provided.

Upvotes: 5

Related Questions