CODEWITHSUNDEEP
kuberneteskubectlkubernetes-ingress
Divyaanand Sinha
Divyaanand Sinha

Reputation: 396

Kubernetes multiple ingress objects with same configs

Suppose I create multiple ingress objects in k8s which point to the same service, same path and are exactly the same, only they have different names eg-. ingress-1 and ingress-2. How are the requests handled in this case? Are the requests duplicated or either ingress handles the request?

PS;- I know this doesn't make much sense but I am testing something out.

Upvotes: 18

Views: 21684

Answers (4)

Konstantin Vustin
Konstantin Vustin

Reputation: 7578

Let's check how it works!

I have default nginx deployment

# kubectl -n test get pods -o wide | grep nginx
nginx-65f88748fd-6w6fj           1/1     Running   0          26h     10.8.253.25    k8s-vm02   <none>           <none>
nginx-65f88748fd-8fp7p           1/1     Running   0          26h     10.8.252.205   k8s-vm01   <none>           <none>
nginx-65f88748fd-c7j29           1/1     Running   0          26h     10.8.253.24    k8s-vm02   <none>           <none>
nginx-65f88748fd-frsbq           1/1     Running   0          26h     10.8.252.201   k8s-vm01   <none>           <none>
nginx-65f88748fd-p4zvm           1/1     Running   0          26h     10.8.252.204   k8s-vm01   <none>           <none>
nginx-65f88748fd-pd8gv           1/1     Running   0          25h     10.8.253.27    k8s-vm02   <none>           <none>
nginx-65f88748fd-rkcjl           1/1     Running   0          26h     10.8.252.206   k8s-vm01   <none>           <none>
nginx-65f88748fd-rn49k           1/1     Running   0          26h     10.8.253.26    k8s-vm02   <none>           <none>
nginx-65f88748fd-w9dz8           1/1     Running   0          26h     10.8.252.203   k8s-vm01   <none>           <none>
nginx-65f88748fd-xh42v           1/1     Running   0          25h     10.8.253.28    k8s-vm02   <none>           <none>

service

# kubectl -n test get svc
NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.8.254.218   <none>        80/TCP    12d

and 2 similar Ingress resources in the same namespace

# kubectl -n test get ing
NAME             HOSTS            ADDRESS   PORTS   AGE
test-ingress-1   nginx.test.com             80      20m
test-ingress-2   nginx.test.com             80      20m

Their YAML's:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: test
  name: test-ingress-1
  annotations:
    kubernetes.io/ingress.class: "service"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: nginx.test.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: nginx
          servicePort: 80

and

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: test
  name: test-ingress-2
  annotations:
    kubernetes.io/ingress.class: "service"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: nginx.test.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: nginx
          servicePort: 80

one more thing about ingress.class: "service" - there are more then 1 ingress controllers in my environment and this particular ingress-controller

nginx-ingress-service-6gkhh                1/1     Running   0          4m20s   10.8.255.243   k8s-vm02   <none>           <none>

has been created specially for demonstrating this example, so pay no attention for it

Anyway, is Ingress resource nginx.test.com/foo working now?

# curl -H "Host: nginx.test.com" http://10.8.255.243:80/foo
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Yes, it is. What's under the hood? Nginx config inside ingress-controller has only one server_name nginx.test.com no matter how many similar Ingress resources we have

# kubectl -n kube-system exec nginx-ingress-service-6gkhh -- cat /etc/nginx/nginx.conf
...
## start server nginx.test.com
    server {
        server_name nginx.test.com ;

        listen 80;

        listen [::]:80;

        set $proxy_upstream_name "-";

        location ~* ^/foo\/?(?<baseuri>.*) {
...
            proxy_pass http://test-nginx-80;
...
    ## end server nginx.test.com

upstreams:

    upstream test-nginx-80 {

        keepalive 32;

        server 10.8.252.203:80 max_fails=0 fail_timeout=0;
        server 10.8.253.26:80 max_fails=0 fail_timeout=0;
        server 10.8.252.201:80 max_fails=0 fail_timeout=0;
        server 10.8.253.25:80 max_fails=0 fail_timeout=0;
        server 10.8.252.204:80 max_fails=0 fail_timeout=0;
        server 10.8.253.24:80 max_fails=0 fail_timeout=0;
        server 10.8.252.205:80 max_fails=0 fail_timeout=0;
        server 10.8.252.206:80 max_fails=0 fail_timeout=0;
        server 10.8.253.27:80 max_fails=0 fail_timeout=0;
        server 10.8.253.28:80 max_fails=0 fail_timeout=0;

    }

Let's delete the test-ingress-1 Ingress resource

# kubectl -n test delete ing test-ingress-1
ingress.extensions "test-ingress-1" deleted

Ingress is still working:

# curl -H "Host: nginx.test.com" http://10.8.255.243:80/foo
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

So, you can create as many similar Ingress resources as you want (In case of using nginx-ingress-controller)

UPD: Let's create one more deployment in the same namespace:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: apache
  template:
    metadata:
      labels:
        app: apache
    spec:
      containers:
      - name: frontend
        image: httpd
        ports:
        - containerPort: 80

and one more service

apiVersion: v1
kind: Service
metadata:
  name: apache
  namespace: test
spec:
  ports:
  - name: 80-80
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: apache
  type: ClusterIP

Next let's change Ingress resource test-ingress-2:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: test
  name: test-ingress-2
  annotations:
    kubernetes.io/ingress.class: "service"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: nginx.test.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: apache # <-------
          servicePort: 80

Finally namespace test includes this list of resources:

# kubectl -n test get all
NAME                                 READY   STATUS    RESTARTS   AGE
pod/apache-cfdf8d79c-2m666           1/1     Running   0          13m
pod/apache-cfdf8d79c-d995s           1/1     Running   0          13m
pod/apache-cfdf8d79c-tq8d7           1/1     Running   0          13m
pod/nginx-65f88748fd-6w6fj           1/1     Running   0          45h
pod/nginx-65f88748fd-8fp7p           1/1     Running   0          45h
pod/nginx-65f88748fd-c7j29           1/1     Running   0          45h
pod/nginx-65f88748fd-frsbq           1/1     Running   0          46h
pod/nginx-65f88748fd-p4zvm           1/1     Running   0          45h
pod/nginx-65f88748fd-pd8gv           1/1     Running   0          45h
pod/nginx-65f88748fd-rkcjl           1/1     Running   0          45h
pod/nginx-65f88748fd-rn49k           1/1     Running   0          45h
pod/nginx-65f88748fd-w9dz8           1/1     Running   0          45h
pod/nginx-65f88748fd-xh42v           1/1     Running   0          45h

NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/apache   ClusterIP   10.8.254.47    <none>        80/TCP    12m
service/nginx    ClusterIP   10.8.254.218   <none>        80/TCP    12d

# kubectl -n test get ing
NAME             HOSTS            ADDRESS   PORTS   AGE
test-ingress-1   nginx.test.com             80      9m15s
test-ingress-2   nginx.test.com             80      54s

the ingreess-controller's nginx.conf shows two lists of upstreams

    upstream test-apache-80 {

        keepalive 32;

        server 10.8.253.31:80 max_fails=0 fail_timeout=0;
        server 10.8.252.208:80 max_fails=0 fail_timeout=0;
        server 10.8.253.32:80 max_fails=0 fail_timeout=0;

    }

    upstream test-nginx-80 {

        keepalive 32;

        server 10.8.252.204:80 max_fails=0 fail_timeout=0;
        server 10.8.252.205:80 max_fails=0 fail_timeout=0;
        server 10.8.253.27:80 max_fails=0 fail_timeout=0;
        server 10.8.253.25:80 max_fails=0 fail_timeout=0;
        server 10.8.253.24:80 max_fails=0 fail_timeout=0;
        server 10.8.252.206:80 max_fails=0 fail_timeout=0;
        server 10.8.253.26:80 max_fails=0 fail_timeout=0;
        server 10.8.252.203:80 max_fails=0 fail_timeout=0;
        server 10.8.253.28:80 max_fails=0 fail_timeout=0;
        server 10.8.252.201:80 max_fails=0 fail_timeout=0;

    }

but it proxies requests only to one of them

    ## start server nginx.test.com
    server {
        server_name nginx.test.com ;

        listen 80;

        listen [::]:80;

        set $proxy_upstream_name "-";

        location ~* ^/foo\/?(?<baseuri>.*) {

            set $namespace      "test";
            set $ingress_name   "test-ingress-1"; <------
            set $service_name   "nginx";
            set $service_port   "80";
            set $location_path  "/foo";

...

            rewrite /foo/(.*) /$1 break;
            rewrite /foo / break;
            proxy_pass http://test-nginx-80;  <-------

            proxy_redirect                          off;

# curl -H "Host: nginx.test.com" http://10.8.255.243:80/foo
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Each of ingress resources appears there in accordance with the creation time. Deleting Ingress resource

ingress.networking.k8s.io "test-ingress-1" deleted

makes ingress-controller proxy traffic to Apache upstreams

# curl -H "Host: nginx.test.com" http://10.8.255.243:80/foo
<html><body><h1>It works!</h1></body></html>

to make it clear we can recreate Ingress resources in reverse order:

# kubectl -n test delete ing test-ingress-1 test-ingress-2
ingress.extensions "test-ingress-1" deleted
ingress.extensions "test-ingress-2" deleted

# kubectl -n test create -f /tmp/ing2.yaml
ingress.networking.k8s.io/test-ingress-2 created # Apache
# kubectl -n test create -f /tmp/ing1.yaml
ingress.networking.k8s.io/test-ingress-1 created # Nginx

# curl -H "Host: nginx.test.com" http://10.8.255.243:80/foo
<html><body><h1>It works!</h1></body></html> # Apache

One more note: There are some cases of reloading the ingress-controller's config, I mean when it reloads config and how it actually reloads config? But that's a totally different question...

Upvotes: 26

Balaji Durairaj
Balaji Durairaj

Reputation: 1

We had a situation where such a configuration was needed in production. And this is how I saw it worked. When you create the second ingress with everything same except just the ingress name, it didn’t affect anything. Only after I deleted the first ingress, the second ingress that I created took over the control. So if the set up is indentical it doesn’t override or duplicate, looks like the first one that was created always has the prio.

Upvotes: 0

Amudhan
Amudhan

Reputation: 725

Ingress controllers (whether it is nginx or haproxy or any other controller) work more or less in the same way. If an Ingress object is created or modified, they listen for that and for every the host/path in the ingress, they create a rule saying that if the request reaches the Ingress controller with that particular host/path, it should forward the traffic to the service that is configured for that path.

So, in your example, you are saying that you are adding multiple Ingress objects having the same host/path configs pointing to the same service. I would think that it is upto the Ingress controller implementation; how they want to handle duplicate Ingress rules. It doesn't make sense to add duplicate rules for the same hostname (doesn't matter whether it points to same service or different service, but in case of different backend service, it is a bigger problem/ambiguity).

But I would assume that even if an Ingress controller allows these kind of configs and programs multiple rules for the same host/path, it wont do multiplexing. Probably it would match the first rule and forward to the service behind it.

How it determines which service to route to? It is based on, on which hostname the traffic reached the Ingress controller.

You can easily check the behavior. Exec into the controller and see its conf file. For nginx ingress controller, the conf file will be in /etc/nginx/nginx.conf which will define the rules and actions.

Upvotes: 1

coderanger
coderanger

Reputation: 54181

If you mean Ingress objects with the same Controller: generally nothing interesting, most of the de-dup redundant routes.

If you mean two Controllers each pointed at a different (but identical) Ingress: both Controllers set themselves up with the requested routes. They are totally independent and don't know anything about each other.

Upvotes: 4

Related Questions