CODEWITHSUNDEEP
grafanagrafana-api
Saikat Chakrabortty
Saikat Chakrabortty

Reputation: 2680

Grafana: Datasource (Prometheus ) query 403

When querying Datasource from grafana getting 403 for a few of the metrics.

curl --location --request GET 'https://xxx.xxx.xxx/api/datasources/proxy/1/api/v1/query?query=sum(kube_pod_container_status_restarts_total%7Bnamespace%3D%22default%22%2C%20container%3D~%22al-agent-container%22%2C%20pod%3D%22al-agent-container-hlrz2%22%7D)&time=1607489911' \
> --header 'Accept: application/json' \
> --header 'Content-Type: application/json; charset=UTF-8' \
> --header 'Authorization: Bearer xxx' \
> --data-raw ''
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>

When Querying from the Prometheus server endpoint with same query it resolves with 200:

curl --location --request GET 'http://kube-system-prometheus-server.kube-system.svc.cluster.local/api/v1/query_range?query=sum(kube_pod_container_resource_requests_cpu_cores%7Bnamespace%3D%22default%22%2C%20pod%3D~%22al-agent-container-hlrz2%22%7D)&start=1607488035&end=1607489835&step=15' \
> --header 'Accept: application/json' \
> --header 'Content-Type: application/json; charset=UTF-8'
{"status":"success","data":{"resultType":"matrix","result":[]}}

Grafana Version: v6.6.2 (3fa63cfc34)

Prometheus does not have any authentication in place, This issue is facing only for few metrics not for all.

When tested the data source from its configuration section. it shows it's all working.

does anyone have any idea on this?

Upvotes: 0

Views: 4444

Answers (3)

Hoiama Rodrigues
Hoiama Rodrigues

Reputation: 346

To resolve issues with Grafana requests being blocked due to large request bodies, follow these steps:

1- Go to the AWS Management Console.
2-Search for WAF and navigate to the Web ACLs section.
3-Locate and select the Web ACL associated with your application.
4-Under Rules, find the AWS-AWSManagedRulesCommonRuleSet rule group.
5-Edit this rule group and locate the rule named SizeRestrictions_BODY.
6-Change its action to Override to Count.

This modification prevents the SizeRestrictions_BODY rule from outright blocking requests with large bodies, such as those generated by Grafana, while still allowing you to monitor their occurrences in logs.

Upvotes: 0

katheravan arumugam
katheravan arumugam

Reputation: 11

From my view , you cant get the data from Grafana. Because Grafana doesn't have DB place to store the data . It will use the Existing database from any other source to plot the Time-based series.

Upvotes: 0

Saikat Chakrabortty
Saikat Chakrabortty

Reputation: 2680

I have found this issue,

I had WAF enabled in the entry LB, I have removed and checked that it does work. WAF is considering this as SQL injection and hence blocking it.

Upvotes: 2

Related Questions