Can't expose Keycloak Server on AWS with Traefik Ingress Controller and AWS HTTPS Load Balancer

Question

I have successfully exposed two microservices on AWS with Traefik Ingress Controller and AWS HTTPS Load Balancer on my registered domain.

Here is the source code: https://github.com/skyglass-examples/user-management-keycloak

I can easily access both microservices with https url:

https://users.skycomposer.net/usermgmt/swagger-ui/index.html
https://users.skycomposer.net/whoami

So, it seems that Traefik Ingress Controller and AWS HTTPS Load Balancer configured correctly.

Unfortunately, Keycloak Server doesn't work in this environment. When I try to access it by https url:

https://users.skycomposer.net/keycloak

I receive the following response:

404 page not found

Do I miss something in my configuration?

Here are some keycloak kubernetes manifests, which I use:

keycloak-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: keycloak
data:
  KEYCLOAK_USER: admin@keycloak
  KEYCLOAK_MGMT_USER: mgmt@keycloak
  JAVA_OPTS_APPEND: '-Djboss.bind.address.management=0.0.0.0'
  PROXY_ADDRESS_FORWARDING: 'true'
  KEYCLOAK_LOGLEVEL: INFO
  ROOT_LOGLEVEL: INFO
  DB_VENDOR: H2

keycloak-deployment.yaml:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: keycloak
  labels:
    app: keycloak

spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
    spec:
      containers:
        - name: keycloak
          image: jboss/keycloak:12.0.4
          imagePullPolicy: Always
          ports:
            - containerPort: 9990
              hostPort: 9990
          volumeMounts:
            - name: keycloak-data
              mountPath: /opt/jboss/keycloak/standalone/data
          env:
            - name: KEYCLOAK_USER
              valueFrom:
                configMapKeyRef:
                  name: keycloak
                  key: KEYCLOAK_USER
            - name: KEYCLOAK_MGMT_USER
              valueFrom:
                configMapKeyRef:
                  name: keycloak
                  key: KEYCLOAK_MGMT_USER
            - name: JAVA_OPTS_APPEND
              valueFrom:
                configMapKeyRef:
                  name: keycloak
                  key: JAVA_OPTS_APPEND
            - name: DB_VENDOR
              valueFrom:
                configMapKeyRef:
                  name: keycloak
                  key: DB_VENDOR
            - name: PROXY_ADDRESS_FORWARDING
              valueFrom:
                configMapKeyRef:
                  name: keycloak
                  key: PROXY_ADDRESS_FORWARDING
            - name: KEYCLOAK_LOGLEVEL
              valueFrom:
                configMapKeyRef:
                  name: keycloak
                  key: KEYCLOAK_LOGLEVEL
            - name: ROOT_LOGLEVEL
              valueFrom:
                configMapKeyRef:
                  name: keycloak
                  key: ROOT_LOGLEVEL
            - name: KEYCLOAK_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: keycloak
                  key: KEYCLOAK_PASSWORD
            - name: KEYCLOAK_MGMT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: keycloak
                  key: KEYCLOAK_MGMT_PASSWORD
      volumes:
        - name: keycloak-data
          persistentVolumeClaim:
            claimName: keycloak-pvc

keycloak-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: keycloak

spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
      targetPort: 9990
  selector:
    app: keycloak

traefik-ingress.yaml:

apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
  name: traefik-lb
spec:
  controller: traefik.io/ingress-controller

---
apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
  name: "traefik-usermgmt-ingress"
spec:
  ingressClassName: "traefik-lb"
  rules:
  - host: "keycloak.skycomposer.net"
    http:
      paths:
      - path: "/usermgmt"
        backend:
          serviceName: "usermgmt"
          servicePort: 80


---
apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
  name: "traefik-whoami-ingress"
spec:
  ingressClassName: "traefik-lb"
  rules:
  - host: "keycloak.skycomposer.net"
    http:
      paths:
      - path: "/whoami"
        backend:
          serviceName: "whoami"
          servicePort: 80


---
apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
  name: "traefik-keycloak-ingress"
spec:
  ingressClassName: "traefik-lb"
  rules:
  - host: "keycloak.skycomposer.net"
    http:
      paths:
      - path: "/keycloak"
        backend:
          serviceName: "keycloak"
          servicePort: 80

See all other files on my github: https://github.com/skyglass-examples/user-management-keycloak

I also checked the logs for keycloak pod, running on my K3S Kubernetes Cluster:

20:57:34,147 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 12.0.4 (WildFly Core 13.0.3.Final) started in 43054ms - Started 687 of 972 services (687 services are lazy, passive or on-demand)
20:57:34,153 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
20:57:34,153 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990

Everything seems to be fine, Admin console is listening on http://127.0.0.1:9990

I also tried using 9990 target port in deployment and service manifests, instead of 8080, but still the same result.

Answer 1

Finally solved the issue.

The following configuation is required to run keycloak behind traefik:

  PROXY_ADDRESS_FORWARDING=true
  KEYCLOAK_HOSTNAME=${YOUR_KEYCLOAK_HOSTNAME}

Also, I had to use the root path "/" for the ingress rule:

apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
  name: "traefik-keycloak-ingress"
spec:
  ingressClassName: "traefik-lb"
  rules:
  - host: "keycloak.skycomposer.net"
    http:
      paths:
      - path: "/"
        backend:
          serviceName: "keycloak"
          servicePort: 80

Here, you can find other configuration properties, which you might find useful: https://github.com/Artiume/docker/blob/master/traefik-SSO.yml

Believe it or not, this is the only resource on the internet, which mentioned KEYCLOAK_HOSTNAME to fix my problem. Two days of searching by keyword "keycloak traefik 404" and no results!

You can find the full fixed code, with correct configuration, on my github: https://github.com/skyglass-examples/user-management-keycloak

Answer 2

I have found one small workaround, but unfortunately, this is not the best solution for me.

I forwarded the port:

kubectl port-forward --address 0.0.0.0 service/keycloak 32080:http

Now Keycloak Server is available on:

http://localhost:32080/auth/

But how to make it available externally by this url ?

https://keycloak.skycomposer.net/keycloak/auth

It is still not clear to me, why the keycloak is not visible from the outside, with my current configuration.

Answer 3

Right - the admin console is listening on 127.0.0.1. This is not the outside world interface. This is "localhost".

You have two choices here. You can start Keycloak with a command line argument like:

bin/standalone.sh -Djboss.bind.address.management=0.0.0.0

This starts the management console on port 9990 but on the 0.0.0.0 interface which is to say all interfaces. So you can still connect to it on localhost but it will now be listening on other (i.e. Ethernet) interfaces.

Another option is to modify the standalone/configuration/standalone.xml file and change:

<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
    </interface>
    <interface name="public">
        <inet-address value="${jboss.bind.address:127.0.0.1}"/>
    </interface>
</interfaces>

to just be:

<interfaces>
    <interface name="management">
        <inet-address value="0.0.0.0"/>
    </interface>
    <interface name="public">
        <inet-address value="${jboss.bind.address:127.0.0.1}"/>
    </interface>
</interfaces>

or whatever address that you'd like Keycloak to listen on. Of course, you can change the public address too if you'd like.

Note that the port is controlled in a different way. The standard way of controlling this is to to run with something like:

bin/standalone.sh -Djboss.socket.binding.port-offset=1000

In this example all ports have 1000 added to them. So the management port went from 9990 to 10990 as 1000 was added to the base.

As a general statement I usually place a proxy (AJP or HTTP) in front of all of my Wildfly servers. That way none of this matters and your proxy connects to, for example, 127.0.0.1, port 9990. But, of course, that's up to you.