CODEWITHSUNDEEP
grafanaprometheus
burk
burk

Reputation: 355

Sometimes getting 405 when using Prometheus from Grafana

I'm using a Prometheus datasource from Grafana, and I'm sometimes getting 200 OK, sometimes getting 405 Method not allowed when looking at graphs, or inserting new graphs.

It is very strange that it only appears sometimes, for random graphs, sometimes only for some graphs in a single dashboard.

The datasource is set up to proxy requests through the backend.

Both Grafana and Prometheus are running in Kubernetes as StatefulSets in Google cloud.

I'm accessing Grafana at localhost:3000 through an SSH tunnel to the pod in Kubernetes, and Grafana is accessing Prometheus at http://prometheus:9090/.

I've tried changing the method from GET to POST in the datasource setup, but then I get 405 on every request.

enter image description here

The raw headers in the request for http://localhost:3000/api/datasources/proxy/1/api/v1/query_range?query=kafka_topic_highwater{topic="test"}&start=1541499015&end=1541499930&step=15 is 

Host: localhost:3000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://localhost:3000/d/tDB6XEaiz/kafka-realtime-timeseries?orgId=1
X-Grafana-Org-Id: 1
DNT: 1
Connection: keep-alive
Cookie: grafana_user=admin; grafana_remember=asdf8a620; grafana_sess=<secret>

And the response is:

HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache
Content-Length: 19
Content-Type: text/plain; charset=utf-8
Date: Tue, 06 Nov 2018 10:25:22 GMT
Expires: -1
Pragma: no-cache
X-Content-Type-Options: nosniff

Any ideas what might be causing this?

Upvotes: 4

Views: 5025

Answers (1)

burk
burk

Reputation: 355

The problem was that I had two Prometheus instances running in the same cluster, with the same service name, so that requests were distributed across them... One of them replied with 405 because it was set up to forward metrics directly to StackDriver..

Upvotes: 3

Related Questions