CODEWITHSUNDEEP
kubernetesmonitoringgrafanaprometheusminikube
online
online

Reputation: 5507

Grafana HTTP Error Bad Gateway and Templating init failed errors

Use helm installed Prometheus and Grafana on minikube at local.

$ helm install stable/prometheus
$ helm install stable/grafana

Prometheus server, alertmanager grafana can run after set port-forward:

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9090

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9093

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=excited-crocodile-grafana,component=grafana" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 3000

enter image description here enter image description here

Add Data Source from grafana, got HTTP Error Bad Gateway error:

enter image description here

Import dashboard 315 from:

https://grafana.com/dashboards/315

Then check Kubernetes cluster monitoring (via Prometheus), got Templating init failed error:

enter image description here

Why?

Upvotes: 13

Views: 29519

Answers (3)

Venryx
Venryx

Reputation: 17999

I was never able to find a "proper" fix, but I found a workaround:

apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  selector:
    app: prometheus
    prometheus: k8s
  sessionAffinity: ClientIP
  clusterIP: None

By setting the clusterIP to None, the service changes to "Headless" mode, which means that requests are sent directly to a random one of the pods in that service/cluster. More info here: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

There's probably a better solution, but this is the only one I've found that actually works for me, with kube-prometheus. (I've tried docker-desktop, k3d, and kind, and all of them have the same issue, so I doubt it's the emulator's fault; and I stripped my config down to basically just kube-prometheus, so it's hard to understand where the problem lies, but oh well.)

Upvotes: 0

Venkataramana Chigateri
Venkataramana Chigateri

Reputation: 1

I turned off the firewall on appliance, post that adding http://prometheus:9090 on URL did not throw bad gateway error.

Upvotes: 0

svenwltr
svenwltr

Reputation: 18442

In the HTTP settings of Grafana you set Access to Proxy, which means that Grafana wants to access Prometheus. Since Kubernetes uses an overlay network, it is a different IP.

There are two ways of solving this:

  1. Set Access to Direct, so the browser directly connects to Prometheus.
  2. Use the Kubernetes-internal IP or domain name. I don't know about the Prometheus Helm-chart, but assuming there is a Service named prometheus, something like http://prometheus:9090 should work.

Upvotes: 22

Related Questions