CODEWITHSUNDEEP
kubernetes-ingress
raja
raja

Reputation: 89

Nginx ingress controller rate limiting not working

  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/limit-connection: "1"
    nginx.ingress.kubernetes.io/limit-rpm: "20"

and the container image version, iam using, image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0

trying to send 200 requests in ten mins of range (and per min it is like a 20 requests from a single ipaddress) and after that it has to refuse the requests.

Upvotes: 1

Views: 3045

Answers (4)

luislhl
luislhl

Reputation: 1506

There are two things that could be making you experience rate-limits higher than configured: burst and nginx replicas.

Burst

As you have already noted in https://stackoverflow.com/a/54426317/3477266, nginx-ingress adds a burst configuration to the final config it creates for the rate-limiting.

The burst value is always 5x your rate-limit value (it doesn't matter if it's a limit-rpm or limit-rps setting.)

That's why you got a burst=100 from a limit-rpm=20.

You can read here the effect this burst have in Nginx behavior: https://www.nginx.com/blog/rate-limiting-nginx/#bursts

But basically it's possible that Nginx will not return 429 for all request you would expect, because of the burst.

The total number of requests routed in a given period will be total = rate_limit * period + burst

Nginx replicas

Usually nginx-ingress is deployed with Horizontal Pod AutoScaler enabled, to scale based on demand. Or it's explicitly configured to run with more than 1 replica.

In any case, if you have more than 1 replica of Nginx running, each one will handle rate-limiting individually.

This basically means that your rate-limit configuration will be multiplied by the number of replicas, and you could end up with rate-limits a lot higher than you expected.

There is a way to use a memcached instance to make them share the rate-limiting count, as described in: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#global-rate-limiting

Upvotes: 0

Vishal Vyas
Vishal Vyas

Reputation: 67

Which nginx ingress version are you using ? please use quay.io/aledbf/nginx-ingress-controller:0.415 and then check, Also Please look at this link - https://github.com/kubernetes/ingress-nginx/issues/1839

Upvotes: 1

raja
raja

Reputation: 89

I changed it to the limit-connections, I am mentioning the annotations in the ingress yml file and applying it and i can in the nginx conf the following 

`worker_rlimit_nofile 15360;
    limit_req_status                503;
    limit_conn_status               503;
    # Ratelimit test_nginx
    # Ratelimit test_nginx `
   ` map $whitelist_xxxxxxxxxxxx $limit_xxxxxxxxxx {
    limit_req_zone $limit_xxxxxxxx zone=test_nginx_rpm:5m rate=20r/m;
            limit_req zone=test_nginx_rpm burst=100 nodelay;
            limit_req zone=test_nginx_rpm burst=100 nodelay;
            limit_req zone=test_nginx_rpm burst=100 nodelay;`


when i kept this annotations, 

 ` nginx.ingress.kubernetes.io/limit-connections: "1" 
  nginx.ingress.kubernetes.io/limit-rpm: "20" `

I can see the above burst and other things in the nginx conf file, can you please tell me these make any differences ?

Upvotes: 0

clxoid
clxoid

Reputation: 2795

Try to change this limit-connection: to limit-connections:

For more info check this

If doesn't help, please put your commands or describe that how are you testing your connection limits.

Upvotes: 0

Related Questions