Using Sidecar Mode for Kubernetes Log Collection

Question

I have deployed my application into a Kubernetes pod along with a fluent-bit sidecar container that collects logs from the sample application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-flb-sidecar
  namespace: default
  labels:
    app.kubernetes.io/name: default
    helm.sh/chart: default-0.1.0
    app.kubernetes.io/instance: flb-sidecar
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/managed-by: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: default
      app.kubernetes.io/instance: flb-sidecar
  template:
    metadata:
      labels:
        app.kubernetes.io/name: default
        app.kubernetes.io/instance: flb-sidecar
    spec:
      containers:
        - name: default
          image: "nginx:stable"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {}
          volumeMounts:
            - name: log-volume
              mountPath: var/log/nginx
        - name: default-fluentbit
          image: "fluent/fluent-bit:1.3-debug"
          imagePullPolicy: IfNotPresent
          ports:
            - name: metrics
              containerPort: 2020
              protocol: TCP
          volumeMounts:
            - name: config-volume
              mountPath: /fluent-bit/etc/
            - name: log-volume
              mountPath: var/log/nginx
      volumes:
        - name: log-volume
          emptyDir: {}
        - name: config-volume
          configMap:
            name: nginx-flb-sidecar

and my fluent-bit is configured to tail logs from /var/log/ngnix/access.log

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-flb-sidecar
  namespace: default
  labels:
    app.kubernetes.io/name: default
    helm.sh/chart: default-0.1.0
    app.kubernetes.io/instance: flb-sidecar
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/managed-by: Tiller
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name tail
        Tag  nginx.access
        Parser nginx
        Path /var/log/nginx/access.log

    [INPUT]
        Name tail
        Tag  nginx.error
        Parser nginx
        Path /var/log/nginx/error.log

    [OUTPUT]
        Name stdout
        Match *

    [OUTPUT]
        Name          forward
        Match         *
        Host          test-l-LoadB-2zC78B5KYFQJC-13137e1aac9bf29c.elb.us-east-2.amazonaws.com
        Port          24224

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*))" "(?<agent>[^\"]*)"(?: "(?<target>[^\"]*))"$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

If I do not have the volume mounts the logs from my application are routed to stdout/stderr.

I need to enable fluent-bit to read from stdout/stderr. How can I acheive this?

Thanks

Answer 1

It's simpler to install fluent-bit as a daemonset rather than sidecar container for several reasons, above all the fact that your container must store the logs in a file that must be standard between all logging pods.

I faced a similar issue and I slightly changed the manifests provided by the fluent-bit installation guide.

In particular I changed the ClusterRole to Role (for RBAC permissions issue) and the volume mounting paths in the daemonset manifest which in my case were different.

Answer 2

To be clear, there's no way to get access to stdout/stderr directly in fluentbit running in kubernetes. You'll need your logs to be written to disk somewhere. In fact, even though it seems to be a bit wasteful, I find that writing to both stdout AND a location on disk is actually better because you get tighter control over the log format and don't have to jump through as many hoops in fluentbit to massage log line into something that works for you (this is great for application logs using a logging provider like log4net or Serilog).

In any case, I thought I'd leave this blurb here since it seems like an approachable solution if you can get logs to stdout AND a location on disk.

At the time of this writing, AWS EKS on Fargate was a little bit "bleeding edge" so we decided to go the sidecar approach since it's a little more feature-rich. Specifically, there's no support for multi-line log messages (which is common when logging exceptions) and there's no support for adding K8s info like pod name, etc through the Kubernetes filter.

In any case, here's a simplified example of my deployment.yml (replace anything surrounded by angle brackets with your stuff.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <appName>
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: <appName>
    spec:
      containers:
      - image: <imageName>
        imagePullPolicy: IfNotPresent
        name: <appName>
        volumeMounts:
        - name: logs
          mountPath: /logs
      - image: public.ecr.aws/aws-observability/aws-for-fluent-bit:2.12.0
        name: fluentbit
        imagePullPolicy: IfNotPresent
        env:
        - name: APP_NAME
          valueFrom: 
            fieldRef:
              fieldPath: metadata.labels['app']
        volumeMounts:
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: logs
          mountPath: /logs
      volumes:
      - name: fluent-bit-config
        configMap:
            name: fluent-bit-config
      - name: logs
        emptyDir: {}

And a simplified version of my configmap.yml (you can create this if you create the fluent-bit.conf and parsers.conf file using the `kubectl create configmap fluent-bit-config --from-file=fluent-bit.conf --from-file=parsers.conf --dry-run=cluent -o yml > configmap.yml). These files end up getting mounted as files under /fluent-bit/etc/ on the running container (which is why I'm referencing parsers.conf in /fluent-bit/etc).

apiVersion: v1
data:
  fluent-bit.conf: |-
    [SERVICE]
        Parsers_File    /fluent-bit/etc/parsers.conf

    [INPUT]
        Name                tail
        Tag                 logs.*
        Path                /logs/*.log
        DB                  /logs/flb_kube.db
        Parser              read_firstline
        Mem_Buf_Limit       100MB
        Skip_Long_Lines     On
        Refresh_Interval    5

    [FILTER]
        Name    modify
        Match   logs.*
        RENAME  log         event
        SET     source      ${HOSTNAME}
        SET     sourcetype  ${APP_NAME}
        SET     host        ${KUBERNETES_SERVICE_HOST}

    [OUTPUT]
        Name          stdout
  parsers.conf: |-
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: fluent-bit-config

Note, one of the clunky parts of this is that any changes to fluentbit configuration require you to force a deployment of your applications because you need the fluentbit sidecar to pick up the new config (you can do so using an annotation with a DateTime or a commit hash, or you could probably even get clever with a readiness probe).

Also note the [FILTER] section. This is where the magic happens with respect to getting kubernetes-contextual-info from the runtime environment (HOSTNAME and KUBERNETES_SERVICE_HOST are provided from K8s and you're injecting the label in your metadata section as APP_NAME). Injecting labels was only added to K8s DownwardAPI in 1.19, so you'll need to be on a newish version.

Answer 3

I don't have any experience with Kubernetes, but in Docker land, you can use the Docker Fluentd log driver. It will forward the stdout/stderr output to fluentd/fluent bit. More information here here

You would have to add/set the input forward in your fluent-bit container.

Answer 4

Looking at supported input plugins it's not possible to configure fluent bit to read logs from stdout/stderr.