Ingress on self-hosted Kubernetes on custom interface

Question

I would like to deploy an ngingx-ingress controller on my self-hosted Kubernetes (microk8s) that is configurable to listen on one or more interfaces (external IPs).

Not even sure if that is easily possible and I should just switch to an external solution, such as HAProxy or an nginx.

Required behavior:

192.168.0.1-H"domain.com":443/frontend -> 192.168.0.1 (eth0) -> ingress -> service-frontend
192.168.0.1-H"domain.com":443/backend -> 192.168.0.1 (eth0) -> ingress -> service-backend
88.88.88.88-H"domain.com":443/frontend -> 88.88.88.88 (eth1) -> ? -> [403 or timeout]
88.88.88.88-H"domain.com":443/backend -> 88.88.88.88 (eth1) -> ? -> [403 or timeout]

And then later the eth1 interface should be able to be switched on, so that requests on that interface behave the same as on eth0.

I would like to be able to deploy multiple instances of services for load-balancing. I would like to keep the configuration in my namespace (if possible) so I can always delete and apply everything at once.

I'm using this guide as a reference: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/

I was able to get something working with minikube, but obviously could not expose any external IPs and performance was quite bad. For that, I just configured a "kind: Ingress" and that was it.

So far, the ingress controller that's default on microk8s seems to listen on all interfaces and I can only configure it in its own namespace. Defining my own ingress seems to not have any effect.

Answer 1

This solution works without enabling ingress on Microk8s:

• install ingress controller : kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/baremetal/deploy.yaml
• create your deployment and service and add this Ingress resource (all in the one namespace):

kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/service-upstream: 'true'
    nginx.ingress.kubernetes.io/rewrite-target: "/$2"
  name: ingress-resource
  namespace: namespace-name
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: service-name
            port:
              number: service-port
        path: /namespace-name/service-name(/|$)(.*)
        pathType: Prefix

• kubectl get svc -n ingress-nginx

• now get either CLUSTER-IP or EXTERNAL-IP and :

  curl ip/namespace-here/service-here

Answer 2

One potential fix was much simpler than anticipated, no messing with MetalLB needed or anything else.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "public"
    nginx.ingress.kubernetes.io/whitelist-source-range: 192.168.0.0/24
...

This does not answer the question of splitting an Ingress across multiple interfaces, but it does solve the problem of restricting public access. By default, bare-metal ingress will listen on all interfaces, which might be a security issue.

Answer 3

I would like to deploy an ngingx-ingress controller on my self-hosted Kubernetes (microk8s) that is configurable to listen on one or more interfaces (external IPs).

For above scenario, you have to deploy the multiple ingress controller of Nginx ingress and keep the different class name in it.

Official document : https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/

So in this scenario, you have to create the Kubernetes service with Loadbalancer IP and each will point to the respective deployment and class will be used in the ingress object.

If you looking forward to use the multiple domains with a single ingress controller you can easily do it by mentioning the host into ingress.

Example for two domain :

1. bar.foo.dev
2. foo.bar.dev

YAML example

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: frontdoor-bar
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  tls:
    - hosts:
        - bar.foo.dev
      secretName: tls-secret-bar
  rules:
    - host: bar.foo.dev
      http:
        paths:
          - backend:
              serviceName: barfoo
              servicePort: 80
            path: /(.*)
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: frontdoor-foo
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - foo.bar.dev
      secretName: tls-secret-foo
  rules:
    - host: foo.bar.dev
      http:
        paths:
          - backend:
              serviceName: foobar
              servicePort: 9000
            path: /(.*)