CODEWITHSUNDEEP
kubernetesprometheuskubernetes-helm
Simon Ernesto Cardenas Zarate
Simon Ernesto Cardenas Zarate

Reputation: 2992

How to configure a prometheus target for kubelet metrics

I would like to plot in Grafana, the metrics for the readiness/liveness probes for some of my pods. Currently, the way I am deploying prometheus in my cluster is using:

helm install prometheus stable/prometheus -n prometheus

I am able to see all standard metrics by going to the prometheus UI, but I am trying to figure out how to get the probes metrics. Apparently the kubelet expose these metrics in /metrics/probes, but I don't know how to configure them. Moreover, I noted that apparently the "standard" metrics are grabbed from the kubernetes api-server on the /metrics/ path, but so far I haven't configured any path nor any config file (I just run the above command to install prometheus). I am assuming that this /metrics/ path is hardcoded somewhere in the helm chart repo, but since I want to get the metrics for the kubelets, this might be trickier, as my understanding is that he api-server lives in the master k8s node, and the kubelet only runs on the worker nodes (so I have no idea where to point the /metrics/probes path).

Upvotes: 1

Views: 2763

Answers (1)

Arghya Sadhu
Arghya Sadhu

Reputation: 44687

Use Prometheus Operator and create ServiceMonitor in which you can specify the endpoints, ports exposed by kubelet or any other component. Prometheus will start scraping the endpoints for metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubelet
  labels:
    k8s-app: kubelet
spec:
  jobLabel: k8s-app
  endpoints:
  - port: https-metrics
    scheme: https
    interval: 30s
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  - port: https-metrics
    scheme: https
    path: /metrics/cadvisor
    interval: 30s
    honorLabels: true
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  selector:
    matchLabels:
      k8s-app: kubelet
  namespaceSelector:
    matchNames:
    - kube-system

Upvotes: 3

Related Questions