In AWS EKS, how can I define ingress to use one ALB for multiple subdomain URLs, each with their own certificate?

Question

I have multiple services that need to be exposed to the internet, but I'd like to use a single ALB for them.

I am using the latest AWS Load Balancer Controller, and I've been reading the documentation here (https://kubernetes-sigs.github.io/aws-load-balancer-controller/guide/ingress/annotations/#traffic-routing), but I haven't found a clear explanation on how to achieve this.

Here's the setup:

I have service-a.example.com -and- service-b.example.com. They each have their own certificates within Amazon Certificate Manager.

Within Kubernetes, each has its own service object defined as follows (each unique):

apiVersion: v1
kind: Service
metadata:
  name: svc-a-service
  annotations:
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthy-threshold-count: '5'
    alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
    alb.ingress.kubernetes.io/healthcheck-path: /index.html
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '30'
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'
    alb.ingress.kubernetes.io/success-codes: '200'
    alb.ingress.kubernetes.io/tags: Environment=Test,App=ServiceA
spec:
  selector:
    app: service-a
  ports:
  - port: 80
    targetPort: 80
  type: NodePort

And each service has it's own Ingress object defined as follows (again, unique to each and with the correct certificates specified for each service):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: svc-a-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: services
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/security-groups: sg-01234567898765432
    alb.ingress.kubernetes.io/ip-address-type: ipv4
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/actions.response-503: >
      {"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"503","messageBody":"Unknown Host"}}
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true,idle_timeout.timeout_seconds=600
    alb.ingress.kubernetes.io/tags: Environment=Test
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:555555555555:certificate/33333333-2222-4444-AAAA-EEEEEEEEEEEE
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-2016-08
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: ssl-redirect
              servicePort: use-annotation
          - path: /*
            backend:
              serviceName: svc-a-service
              servicePort: 80
          - path: /*
            backend:
              serviceName: response-503
              servicePort: use-annotation

The HTTP to HTTPS redirection works as expected.

However -- there is no differentiation between my two apps for the load balancer to be able to know that traffic destined for service-a.example.com and service-b.example.com should be routed to two different target groups.

In the HTTP:443 listener rules in the console, it shows:

1. IF Path is /* THEN Forward to ServiceATargetGroup
2. IF Path is /* THEN Return fixed 503
3. IF Path is /* THEN Forward to ServiceBTargetGroup
4. IF Path is /* THEN Return fixed 503
5. IF Request otherwise not routed THEN Return fixed 404

So the important question here is: How should the ingress be defined to force traffic destined for service-a.example.com to ServiceATargetGroup - and traffic destined for service-b.example.com to ServiceBTargetGroup?

And secondarily, I need the "otherwise not routed" to return a 503 instead of 404. I was expecting this to appear only once in the rules (be merged) - yet it is created for each ingress. How should my yaml be structured to achieve this?

Answer 1

AWS EKS now has a notion of IngressGroups so multiple ingresses can share one ingress controller. See Application load balancing on Amazon EKS

To share an application load balancer across multiple ingress resources using IngressGroups

To join an Ingress to an Ingress group, add the following annotation to a Kubernetes Ingress resource specification.

alb.ingress.kubernetes.io/group.name: <my-group>
The group name must be:

63 characters or less in length.

Consist of lower case alphanumeric characters, -, and ., and must start and end with an alphanumeric character.

The controller will automatically merge ingress rules for all Ingresses in the same Ingress group and support them with a single ALB. Most annotations defined on an Ingress only apply to the paths defined by that Ingress. By default, Ingress resources don't belong to any Ingress group.

Answer 2

I eventually figured this out -- so for anyone else stumbling onto this post, here's how I resolved it:

The trick was not relying on merging between the Ingress objects. Yes, it can handle a certain degree of merging, but there's not really a one-to-one relationship between Services as TargetGroups and Ingress as ALB. So you have to be very cautious and aware of what's in each Ingress object.

Once I combined all of my ingress into a single object definition, I was able to get it working exactly as I wanted with the following YAML:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: svc-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: services
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/security-groups: sg-01234567898765432
    alb.ingress.kubernetes.io/ip-address-type: ipv4
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/actions.response-503: >
      {"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"503","messageBody":"Unknown Host"}}
    alb.ingress.kubernetes.io/actions.svc-a-host: >
      {"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"svc-a-service","servicePort":80,"weight":100}]}}
    alb.ingress.kubernetes.io/conditions.svc-a-host: >
      [{"field":"host-header","hostHeaderConfig":{"values":["svc-a.example.com"]}}]
    alb.ingress.kubernetes.io/actions.svc-b-host: >
      {"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"svc-b-service","servicePort":80,"weight":100}]}}
    alb.ingress.kubernetes.io/conditions.svc-b-host: >
      [{"field":"host-header","hostHeaderConfig":{"values":["svc-b.example.com"]}}]
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true,idle_timeout.timeout_seconds=600
    alb.ingress.kubernetes.io/tags: Environment=Test
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:555555555555:certificate/33333333-2222-4444-AAAA-EEEEEEEEEEEE,arn:aws:acm:us-east-2:555555555555:certificate/44444444-3333-5555-BBBB-FFFFFFFFFFFF
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-2016-08
spec:
  backend:
    serviceName: response-503
    servicePort: use-annotation
  rules:
    - http:
        paths:
          - backend:
              serviceName: ssl-redirect
              servicePort: use-annotation
          - backend:
              serviceName: svc-a-host
              servicePort: use-annotation
          - backend:
              serviceName: svc-b-host
              servicePort: use-annotation

Default Action:

Set by specifying the serviceName and servicePort directly under spec:

spec:
  backend:
    serviceName: response-503
    servicePort: use-annotation

Routing:

Because I'm using subdomains and paths won't work for me, I simply omitted the path and instead relied on hostname as a condition.

metadata:
  alb.ingress.kubernetes.io/actions.svc-a-host: >
      {"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"svc-a-service","servicePort":80,"weight":100}]}}
  alb.ingress.kubernetes.io/conditions.svc-a-host: >
      [{"field":"host-header","hostHeaderConfig":{"values":["svc-a.example.com"]}}]

End Result:

The ALB rules were configured precisely how I wanted them:

• default action is a 503 fixed response
• all http traffic is redirected to https
• traffic is directed to TargetGroups based on the host header