CODEWITHSUNDEEP
kubernetesgoogle-kubernetes-engineprometheus
MatsuzakaSteven
MatsuzakaSteven

Reputation: 317

Prometheus getting 403 forbidden from kubernetes api in GKE

For the prometheus deployment's ClusterRole I have

# ClusterRole for the deployment
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

With the ServiceAccount and the ClusterRoleBinding already put to place too.

And the following are the settings for the jobs inside prometheus.yml that are getting 403 error

- job_name: 'kubernetes-cadvisor'

      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

- job_name: 'kubernetes-nodes'

      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

I don't get the reason why I keep getting 403 error even though the ServiceAccount and the ClusterRole has been binded together.

Upvotes: 0

Views: 4187

Answers (1)

TheHakky
TheHakky

Reputation: 110

Make sure that the /var/run/secrets/kubernetes.io/serviceaccount/token file contains the correct token. To do so, you can enter into Prometheus pod with:

kubectl exec -it -n <namespace> <Prometheus_pod_name> -- bash

and cat the token file. Then exit the pod and execute:

echo $(kubectl get secret -n <namespace> <prometheus_serviceaccount_secret> -o jsonpath='{.data.token}') | base64 --decode

If the tokens match, you can try querying the Kubernetes API server with Postman or Insomnia to see if the rules you put in your ClusterRole are correct. I suggest you to query both /proxy/metrics/cadvisor and /proxy/metrics URLs

Upvotes: 4

Related Questions