Kubernetes ingress same path multiple ports

Question

After much googling and searching (even here), I'm not able to find a definitive answer to my question. So I hope someone here might be able to point me in the right direction.

I have a Kube Service definition that's already working for me, but right now I've simply exposed it with just a LoadBalancer. Here's my current Service yaml:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: namespace1
  labels:
    app: my-service
spec:
  type: LoadBalancer
  selector:
    app: my-service
    tier: web
  ports:
  - name: proxy-port
    port: 8080
    targetPort: 8080
  - name: metrics-port
    port: 8082
    targetPort: 8082
  - name: admin-port
    port: 8092
    targetPort: 8092
  - name: grpc-port
    port: 50051
    targetPort: 50051

This is obviously only TCP load-balanced. What I want to do is secure this with Mutual TLS, so that the server will only accept connections from my client with the authorized certificate.

From all I can tell in Kube land, what I need to do that is an Ingress definition. I've been researching all the docs I can find on kind:Ingress and I can't seem to find anything where it allows me to create a single Ingress with multiple ports on the same path!

Am I missing something here? Is there no way to create a K8s Ingress that simply has the same functionality as the above Service definition?

Answer 1

I faced the same situation where we had to expose port 80,443 and 50051 on the same host. Using traefik v2+ on K3S, this is how I solved it:

Apply this file to modify traefik config; this is with K3S. If you have installed traefik directly with the chart, add a values file with the same config as below.

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
   ports:
      grpc:
        port: 50051
        protocol: TCP
        expose: true
        exposedPort: 50051

After it is done, watch the traefik service be updated with the new config. If its not working, make sure the port youve set are free and not used by another service.

When this is done, you can create IngressRoute object. Here I got one for grpc (50051) and one for web (80/443).

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: db
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - kind: Rule
      match: Host(`lc1.nebula.global`)
      services:
        - name: db
          port: 80

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: db-grpc
spec:
  entryPoints:
    - grpc
  routes:
    - kind: Rule
      match: Host(`lc1.nebula.global`)
      services:
        - name: db-grpc
          port: 50051

EDIT::: if running with your own instance of k3s with the community helm chart (not the one provided by k3s). Here is the equivalent config I have:

traefik: 
  rbac:
    enabled: true
  ports:
    web:
      redirectTo: websecure
    websecure:
      tls:
        enabled: true
    grpc:
      port: 50051
      protocol: TCP
      expose: true
      exposedPort: 50051
  podAnnotations:
    prometheus.io/port: "8082"
    prometheus.io/scrape: "true"
  providers:
    kubernetesIngress:
      publishedService:
        enabled: true
  priorityClassName: "system-cluster-critical"
  tolerations:
  - key: "CriticalAddonsOnly"
    operator: "Exists"
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
  additionalArguments:
    - "--entrypoints.grpc.http2.maxconcurrentstreams=10000"

ingress:
  enabled: false
  host: traefik.local
  annotations: {}

In my case, I also increase the max number of concurrent http2 streams for grpc.

Answer 2

To my knowledge you cannot use custom ports (e.g 8080) for HTTPS LoadBalancer backed with Ingress Controller (e.g. NGINX HTTP(S) Proxy), as currently the port of an Ingress is implicitly :80 for http and :443 for https, as official doc reference for IngressRule explains.

I think the workaround would be to use different host per service, like with this example of Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: name-virtual-host-ingress
spec:
  rules:
  - host: proxy.foo.com
    http:
      paths:
      - backend:
          serviceName: proxy-svc
          servicePort: 8080
  - host: metrics.foo.com
    http:
      paths:
      - backend:
          serviceName: metrics-svc
          servicePort: 8082
  - host: admin.foo.com
    http:
      paths:
      - backend:
          serviceName: admin-svc
          servicePort: 8092
  - host: grpc.foo.com
    http:
      paths:
      - backend:
          serviceName: grpc-svc
          servicePort: 50051