CODEWITHSUNDEEP
nginxkubernetesprometheusnginx-ingress
wilfo
wilfo

Reputation: 715

How to properly log the "Path" in K8S ingress-nginx metrics

I'm using ingress-nginx as an Ingress controller for one of my services running over K8S (I'm using the nginx-0.20.0 release image with no specific metrics configurations in the K8S configmap the ingress-controller is using).

The nginx-ingress-controller pods are successfully scraped into my Prometheus server but all ingress metrics (e.g. nginx_ingress_controller_request_duration_seconds_bucket) show up with path="/" regardless of the real path of the handled request.

Worth noting that when I look at the ingress logs - the path is logged correctly.

How can I get the real path noted in the exported metrics?

Thanks!

Upvotes: 5

Views: 4360

Answers (2)

wilfo
wilfo

Reputation: 715

The Path attribute in the NGINX metrics collected by prometheus derives from the Ingress definition yaml.

For example, if your ingress is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: <some-k8s-ingress-name>
  namespace: <some-k8s-namespace-name>
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          serviceName: <some-k8s-service-name>
          servicePort: <some-port>
        path: /

Then although NGINX will match any URL to your service, it'll all be logged under the path "/" (as seen here).

If you want metrics for a specific URL, you'll need to explicitly specify it like this (notice the ordering of rules):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 annotations:
   kubernetes.io/ingress.class: nginx
 name: <some-k8s-ingress-name>
 namespace: <some-k8s-namespace-name>
spec:
 rules:
 - host: <hostname>
   http:
     paths:
     - backend:
         serviceName: <some-k8s-service-name>
         servicePort: <some-port>
       path: /more/specific/path
     - backend:
         serviceName: <some-k8s-service-name>
         servicePort: <some-port>
       path: /

Upvotes: 5

evgenyl
evgenyl

Reputation: 8107

If I am understand right, this label was removed as causing high cardinality: https://github.com/kubernetes/ingress-nginx/issues/2924#issuecomment-424516095

And this is more general issue: https://github.com/kubernetes/ingress-nginx/pull/2701

Have to say that I am pretty agree - think about it: if you allow the route to be the label value, it can be pretty easy to DDoS your Prometheus - just call your service with a lot of unexisting routes.

All those calls will be translated into the Label Values for all ingress metrics....

Upvotes: 1

Related Questions