CODEWITHSUNDEEP
amazon-web-servicesloggingkubernetesamazon-cloudwatchfluent-bit
Shinebayar G
Shinebayar G

Reputation: 5180

Fluent-Bit sending kubernetes logs to AWS Cloudwatch. How to change log stream prefix based on kubernetes namespace?

I'm using Fluent-Bit to ship kubernetes container logs into cloudwatch. This config is working fine. Instead of output-elasticsearch.conf I have following:

output-cloudwatch.conf: |
    [OUTPUT]
        Name                cloudwatch_logs
        Match               *
        region              us-east-1
        log_group_name      /aws/eks/eks-cluster-1234/containers
        log_stream_prefix   <kubernetes-namespace>
        auto_create_group   On

How can I grab the kubernetes namespace value for this config? So our cloudwatch logs will be little bit organized.

Thank you.

Upvotes: 3

Views: 1516

Answers (1)

Zhyer
Zhyer

Reputation: 11

I had a same issue and I used this raw file to extract the necessary Application.* inputs and filters that would allow You to use $(tag['0') as a log_stream key.

https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-compatible.yaml

So once You enable the Kubernetes filter and the necessary filters You will be able to set Your log_stream_name in the [OUTPUT]: $(kubernetes['container_name'])

Or in Your case log_stream_prefix: $(kubernetes['namespace'])

Hope this helps.

Edit:

I should also mention, that in order to use the tags You need a fluent-bit cloudwatch plugin. Because I assumed by default that You are using a fluent-bit image with the latest cloudwatch plugin already in it. In case this does not work and it turns out You do not have the plugin, here is the link with the fluent-bit image that has it included from their official ecr repository:

https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit

Upvotes: 1

Related Questions