CODEWITHSUNDEEP
kubernetesprometheus
JasonW
JasonW

Reputation: 453

Prometheus cannot find Apiserver of self-hosted Kubernetes cluster

I set up a Kubernetes cluster. Apiserver is started on host 192.168.0.2, and I use self-signed certificate and static token as authentication. The other 2 nodes' ip are 192.168.0.3 and 192.168.0.4

Then I created a prometheus deployment, the config is this. In prometheus dashboard, the two nodes and prometheus self are showed as "UP". However the apiserver is showed "DOWN", and the reason is "context deadline exceeded". 

kubernetes-apiservers
Endpoint    State   Labels  Last Scrape Error
https://192.168.0.2:443/metrics
DOWN    instance="192.168.0.2:443"  55.979s ago context deadline exceeded

I tried to curl the address (https://192.168.0.2:443/metrics) from node server with ca certificate and token. The result is ok.

By the way, I setup the apiserver by hyperkube apiserver command line instead of kubelet. Is this the problem? And where should I look into?

In prometheus log, I only found this related one.

time="2017-01-13T10:51:28Z" level=debug msg="endpoints update" kubernetes_sd=endpoint source="endpoints.go:77" tg="&config.TargetGroup{Targets:[]model.LabelSet{model.LabelSet{\"__meta_kubernetes_endpoint_port_protocol\":\"TCP\", \"__meta_kubernetes_endpoint_ready\":\"true\", \"__address__\":\"192.168.0.2:443\", \"__meta_kubernetes_endpoint_port_name\":\"https\"}}, Labels:model.LabelSet{\"__meta_kubernetes_service_label_provider\":\"kubernetes\", \"__meta_kubernetes_namespace\":\"default\", \"__meta_kubernetes_endpoints_name\":\"kubernetes\", \"__meta_kubernetes_service_name\":\"kubernetes\", \"__meta_kubernetes_service_label_component\":\"apiserver\"}, Source:\"endpoints/default/kubernetes\"}"

Update:

The reason is that I didn't set up calico on master node. It works now.

Upvotes: 1

Views: 742

Answers (1)

brian-brazil
brian-brazil

Reputation: 34112

That sounds like a network issue as the request is timing out. Can you hit that endpoint from inside the Prometheus container?

Upvotes: 2

Related Questions