ETCD certs data to Prometheus

Question

I'm trying to get my head around how to get prometheus https://hub.helm.sh/charts/stable/prometheus collect etcd stats. I understand I need to set tls for it, but have a hard time to find good way to do it without manual additional ansible steps. Is there the way I can get etcd certs on worker node and mount them to prometheus pod?

Answer 1

In addition to https://stackoverflow.com/a/58876108/27730930 (point 5)

If you use IAC or GitOps approach you can deliver etcd certificates to kubernetes secret with standard k8s mechanisms:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-etcd-certs-to-secret-egress-to-apiserver
  namespace: tech-monitoring
spec:
  podSelector:
    matchLabels:
      app: etcd-certs-to-secret
  policyTypes:
  - Egress
  egress:
  - to:
      namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          component: kube-apiserver
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: etcd-certs-to-secret
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: etcd-certs-to-secret
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: etcd-certs-to-secret
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: etcd-certs-to-secret
subjects:
- kind: ServiceAccount
  name: etcd-certs-to-secret
---
apiVersion: batch/v1
kind: Job
metadata:
  name: etcd-certs-to-secret
spec:
  template:
    metadata:
      labels:
        app: etcd-certs-to-secret
    spec:
      serviceAccountName: etcd-certs-to-secret
      containers:
      - name: apply-secret
        securityContext:
          runAsUser: 0
          runAsGroup: 0
        image: bitnami/kubectl:1.32.0
        command: ["/bin/sh", "-c"]
        args:
          - |
            if [ ! -f /etcd-certs/ca.crt ]; then
              echo "Error: Certificate authority file '/etcd-certs/ca.crt' is missing."
              exit 1
            fi

            if [ ! -f /etcd-certs/tls.crt ]; then
              echo "Error: Certificate file '/etcd-certs/tls.crt' is missing."
              exit 1
            fi

            if [ ! -f /etcd-certs/tls.key ]; then
              echo "Error: Key file '/etcd-certs/tls.key' is missing."
              exit 1
            fi

            kubectl apply -f - <<EOF
            apiVersion: v1
            kind: Secret
            metadata:
              name: etcd-certs
              annotations:
                created-by: job/etcd-certs-to-secret
            type: kubernetes.io/tls
            data:
              ca.crt: $(cat /etcd-certs/ca.crt | base64 -w 0)
              tls.crt: $(cat /etcd-certs/tls.crt | base64 -w 0)
              tls.key: $(cat /etcd-certs/tls.key | base64 -w 0)
            EOF
        volumeMounts:
        - name: ca-crt
          mountPath: /etcd-certs/ca.crt
          readOnly: true
        - name: tls-crt
          mountPath: /etcd-certs/tls.crt
          readOnly: true
        - name: tls-key
          mountPath: /etcd-certs/tls.key
          readOnly: true
      restartPolicy: Never
      tolerations:
      - key: "node-role.kubernetes.io/control-plane"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      volumes:
      - name: ca-crt
        hostPath:
          path: /etc/kubernetes/pki/etcd/ca.crt
          type: File
      - name: tls-crt
        hostPath:
          path: /etc/kubernetes/pki/etcd/server.crt
          type: File
      - name: tls-key
        hostPath:
          path: /etc/kubernetes/pki/etcd/server.key
          type: File
  backoffLimit: 4

Answer 2

Following the Monitoring External Etcd Cluster With Prometheus Operator you can easily configure Prometheus to scrape metrics from ETCD.

We can do all of that by creating certs as kubernetes secrets and adding a tlsConfig to our service monitor. Let me walk you through the whole process.

The steps are:

1) Create etcd service

2) Create/attach endpoints for etcd service

3) Create service monitor with appropriate tlsconfig. below example

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: etcd
  name: etcd
  namespace: kube-system
spec:
  endpoints:
  - interval: 30s
    port: metrics
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client-ca.crt
      certFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.crt
      keyFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.key
      serverName: etcd-cluster
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: etcd

4) Create Etcd Client Certificates

5) Create Kubernetes Secrets along with previously created certificate and key for prometheus and etcd ca. This will allow prometheus to securely connect to etcd. Example:

kubectl -n monitoring create secret kube-etcd-client-certs --from-file=etcd-client-ca.crt=etcd-client.ca.crt --from-file=etcd-client.crt=etcd-client.crt --from-file=etcd-client.key=etcd-client.key

6) Update prometheus.yaml to include there names of the created secrets.

7) delploy etcd-service,servicemonitor and prometheus manifests to cluster

kubectl apply -f etcd-service.yaml
kubectl apply -f etcd-serviceMon.yaml
kubectl apply -f prometheus-prometheus.yaml

Enjoy