Reputation: 111
I got an alert while configuring the monitoring module using prometheus/kube-prometheus-stack 25.1.0
.
Alert
[FIRING:1] KubeProxyDown - critical
Alert: Target disappeared from Prometheus target discovery. - critical
Description: KubeProxy has disappeared from Prometheus target discovery.
Details:
• alertname: KubeProxyDown
• prometheus: monitoring/prometheus-kube-prometheus-prometheus
• severity: critical
I think it is a new default rule in kube-prometheus-stack 25.x.x
. It does not exist in prometheus/kube-prometheus-stack 21.x.x
.
The same issue happened in the EKS and minikube.
KubeProxyDown Rule
alert: KubeProxyDown
expr: absent(up{job="kube-proxy"}
== 1)
for: 15m
labels:
severity: critical
annotations:
description: KubeProxy has disappeared from Prometheus target discovery.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubeproxydown
summary: Target disappeared from Prometheus target discovery.
How can I resolve this issue?
I would be thankful if anyone could help me
Upvotes: 8
Views: 8009
Reputation: 1893
Both previous answers are correct, but if you are upgrading to eks 1.22, you could only upgrade the kube-proxy addon to v1.22.11-eksbuild.2 (current one) and the cm will be automatically updated
metricsBindAddress: 127.0.0.1:10249
to
metricsBindAddress: 0.0.0.0:10249
without the need to update it manually
You can see AWS kube-proxy addon documentation from https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html
Upvotes: 0
Reputation: 7796
This is what worked for me in AWS EKS cluster v1.21:
$ kubectl edit cm/kube-proxy-config -n kube-system
---
metricsBindAddress: 127.0.0.1:10249 ### <--- change to 0.0.0.0:10249
$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system
Note, the name of the config map is kube-proxy-config
, not kube-proxy
Upvotes: 5
Reputation: 522
There was a change in metrics-bind-address
in kube-proxy
. Following the issues posted here, here and here. I can suggest the following. Change kube-proxy
ConfigMap to different value:
$ kubectl edit cm/kube-proxy -n kube-system
## Change from
metricsBindAddress: 127.0.0.1:10249 ### <--- Too secure
## Change to
metricsBindAddress: 0.0.0.0:10249
$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system
Upvotes: 5