CODEWITHSUNDEEP
google-kubernetes-engineprometheusprometheus-operatorkube-proxykube-prometheus-stack
Sameer Malhotra
Sameer Malhotra

Reputation: 464

Getting KubeControllerManager, KubeProxy, KubeScheduler down alert in Kube Prometheus Stack installed in GKE

I just installed the latest kube prometheus stack (kube-prometheus-stack-37.2.0) with default setting in my GKE cluster.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring

I started getting three alerts (Getting KubeControllerManager, KubeProxy, KubeScheduler down). By doing some research I found that I need to change the kube proxy's metricsBindAddress to 0.0.0.0:10249 in ConfigMap. But I can't find any kube-proxy or kube-proxy-config in kube-system namespace. Not sure what to do to fix this issue.

Upvotes: 1

Views: 4505

Answers (2)

Granti4ka
Granti4ka

Reputation: 9

SOLUTION: You just need to specify the right ports and additional options for both components. And everything will be fine and works!

Add the next directives to your values files:

## Component scraping the kube controller manager
##
kubeControllerManager:
  service:
    enabled: true
    ports:
      http: 10257
    targetPorts:
      http: 10257
  serviceMonitor:
    https: true
    insecureSkipVerify: "true"

## Component scraping kube proxy
##
kubeProxy:
  enabled: false
  service:
    enabled: true
    port: 10249
    targetPort: 10249

## Component scraping kube scheduler
##
kubeScheduler:
  service:
    enabled: true
    ports:
      http: 10259
    targetPorts:
      http: 10259
  serviceMonitor:
    https: true
    insecureSkipVerify: "true"

Upvotes: 0

Sameer Malhotra
Sameer Malhotra

Reputation: 464

I couldn't find any answer on how to fix this issue of prometheus not being able to connect to KubeControllerManager, KubeProxy and KubeScheduler in GKE. So had to disable it for now in prometheus. Here is how I did it in case this helps someone else.

Create a yaml file (kube-prometheus-stack-overrides.yaml). Add the below content to this file:

## Component scraping the kube controller manager
##
kubeControllerManager:
  enabled: false
## Component scraping kube proxy
##
kubeProxy:
  enabled: false
  service:
    enabled: true
    port: 10249
    targetPort: 10249
## Component scraping kube scheduler
##
kubeScheduler:
  enabled: false

So basically, this file tells prometheus not to worry about these components (this is probably not a good idea but till we find a better solution, this will do!).

Execute the below command

helm upgrade -f kube-prometheus-stack-overrides.yaml [release name] prometheus-community/kube-prometheus-stack -n [namespace where stack is installed]

Upvotes: 0

Related Questions