CODEWITHSUNDEEP
azurekubernetesazure-blob-storagefluentdazure-aks
vishal
vishal

Reputation: 1874

Why is Fluentd Azure blob plugin not working in kubernetes

I've installed fluentd in my AKS cluster using the following command helm install fluentd bitnami-azure/fluentd --namespace mynamespace --set forwarder.configMap=fluentd-aksconfig.

Below is my configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-aksconfig
  namespace: mynamespace
data:
  fluentd.conf: |
    <match **>
          @type azure-storage-append-blob
          azure_storage_account    mystorageaccount
          azure_storage_access_key myaccesskey
          azure_container          aksfluentd
          auto_create_container    true
          path logs/
          azure_object_key_format  %{path}%{time_slice}_%{index}.log
          time_slice_format        %Y%m%d-%H
          # if you want to use %{tag} or %Y/%m/%d/ like syntax in path / azure_blob_name_format,
          # need to specify tag for %{tag} and time for %Y/%m/%d in <buffer> argument.
          <buffer tag,time>
            @type file
            path /var/log/fluent/azurestorageappendblob
            timekey 120 # 2 minutes
            timekey_wait 60
            timekey_use_utc true # use utc
          </buffer>
    </match>

below are the errors obtained from the pods. It displays that there is no plugin called "azure-storage-append-blob". How to fix this?

Logs:

[38;5;6mfluentd [38;5;5m10:03:45.15 [0m [38;5;6mfluentd [38;5;5m10:03:45.15 [0m[1mWelcome to the Bitnami fluentd container[0m [38;5;6mfluentd [38;5;5m10:03:45.16 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-fluentd[0m [38;5;6mfluentd [38;5;5m10:03:45.16 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-fluentd/issues[0m [38;5;6mfluentd [38;5;5m10:03:45.16 [0mSend us your feedback at [[email protected][0m [38;5;6mfluentd [38;5;5m10:03:45.16 [0m [38;5;6mfluentd [38;5;5m10:03:45.16 [0m[38;5;2mINFO [0m ==> ** Starting Fluentd setup ** [38;5;6mfluentd [38;5;5m10:03:45.38 [0m[38;5;2mINFO [0m ==> ** Fluentd setup finished! ** [38;5;6mfluentd [38;5;5m10:03:45.39 [0m[38;5;2mINFO [0m ==> ** Starting Fluentd ** 2020-02-18 10:03:46 +0000 [info]: parsing config file is succeeded path="/opt/bitnami/fluentd/conf/fluentd.conf" 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0' 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13' 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.1' 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-kafka' version '0.12.2' 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.4.2' 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0' 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.7.0' 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-s3' version '1.2.1' 2020-02-18 10:03:46 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2' 2020-02-18 10:03:46 +0000 [info]: gem 'fluentd' version '1.9.1' 2020-02-18 10:03:46 +0000 [error]: config error file="/opt/bitnami/fluentd/conf/fluentd.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'azure-storage-append-blob'. Run 'gem search -rd fluent-plugin' to find plugins"

Upvotes: 0

Views: 1922

Answers (2)

vishal
vishal

Reputation: 1874

As said by @ArghyaSadhu in his answer, Currently fluentd docker image does not come with azure blob plugin pre-installed. We can obtain the docker image and customize it. Please refer to the following link on how to customize fluentd's docker image. https://github.com/bitnami/bitnami-docker-fluentd#customize-this-image

Upvotes: 0

Arghya Sadhu
Arghya Sadhu

Reputation: 44559

azure-storage-append-blob plugin is not installed by default. So you need to install it. Docs on how to install Fluentd plugin is here.

Upvotes: 3

Related Questions