Can't communicate with Elasticsearch endpoint from FluentBit

Question

Problem:

Connection by AWS Elasticsearch endpoint is refused when pushing Kubernetes logs through a fluentBit forwarder.

Here is the fluentBit set up:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        K8S-Logging.Parser  On
  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Retry_Limit     False
        tls             Off
        tls.verify      Off
#----
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.5
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "https://vpc-tf-test2-xyzxyzxyzxyz.eu-west-2.es.amazonaws.com"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "443"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

The fluentBit forwarder logs show this:

[2021/02/01 09:09:11] [error] [io] connection #46 failed to: https://vpc-tf-test2-xyzxyzxyz.eu-west-2.es.amazonaws.com:443
[2021/02/01 09:09:11] [ warn] [engine] failed to flush chunk '1-1611849613.623397482.flb', retry in 1521 seconds: task_id=1980, input=tail.0 > output=es.0
[2021/02/01 09:09:11] [ warn] [engine] failed to flush chunk '1-1611849347.548817423.flb', retry in 1806 seconds: task_id=1623, input=tail.0 > output=es.0
[2021/02/01 09:09:11] [ warn] [engine] failed to flush chunk '1-1611849095.485002520.flb', retry in 1286 seconds: task_id=1284, input=tail.0 > output=es.0
[2021/02/01 09:09:13] [ warn] net_tcp_fd_connect: getaddrinfo(host='https://vpc-tf-test2-xyzxyzxyzxyz.eu-west-2.es.amazonaws.com'): Name or service not known
[2021/02/01 09:09:13] [error] [io] connection #46 failed to: https://vpc-tf-test2-xyzxyzxyz.eu-west-2.es.amazonaws.com:443
[2021/02/01 09:09:13] [ warn] [engine] failed to flush chunk '1-1611849450.549250742.flb', retry in 799 seconds: task_id=1766, input=tail.0 > output=es.0

I am trying to trace where the access is getting blocked.

Access to the ES endpoint is protected by Security Group with this inbound rules:

Type:  All traffic                     
Protocol: All
Port range: All
Source: sg-xyzxyzxyz (eks-cluster-sg-vrs2-eks-dev-xyzxyzyxz)

Answer 1

Change FLUENT_ELASTICSEARCH_HOST value to vpc-tf-test2-xyzxyzxyzxyz.eu-west-2.es.amazonaws.com.