Error Too many redirects when replicas of container secured by keycloak are higher than 1

Question

I'm setting up a docker swarm with the following services: * nginx (acting as reverse proxy)[docker version alpine-14] * wildfly (serving my secured app) * keycloak (securing my app)[docker version Keycloak4.0.0.Final]

Everything goes fine and I can authenticate and access my app when I have only one replica of my app. BUT when I try to scale my wildfly service to more than 1 replica, I can access the login page and once credentials are introduced it gives the error ERR_TOO_MANY_REDIRECTS.

I have tried to change my nginx proxy configuration to forward requests to keycloak https and http ports, in the keycloak side I tried to add

  environment:
   PROXY_ADDRESS_FORWARDING: "true"

Almost everything is working when I only have one replica of my wildfly service.. but same error keeps appearing when having > 1 replicas of it.

This is my nginx config file:

server {
    listen 443 ssl;
    ssl on;
    ssl_certificate /etc/ssl/fullchain.pem;        # path to your cacert.pem
    ssl_certificate_key /etc/ssl/privkey.pem;    # path to your privkey.pem
    server_name testsite.com;

    rewrite     ^/$ /web/ permanent;

   location / {
        proxy_pass         http://wildfly.service.com:8080/;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
        proxy_set_header   X-Forwarded-Proto https;

        # to avoid upstream sent too big header while reading response header from upstream ERROR
        # thanks to https://ma.ttias.be/nginx-proxy-upstream-sent-big-header-reading-response-header-upstream/
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;

        proxy_pass_header  Set-Cookie;
    }

    location /auth {
        # proxy_pass         http://keycloak.service.com:8080/auth;
        proxy_pass         https://keycloak.service.com:8443/auth;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
        proxy_set_header   X-Forwarded-Proto https;

        # to avoid upstream sent too big header while reading response header from upstream ERROR
        # thanks to https://ma.ttias.be/nginx-proxy-upstream-sent-big-header-reading-response-header-upstream/
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;

        proxy_pass_header  Set-Cookie;
    }
}

My keycloak config:

Enabled                 ON
Consent Required            OFF
Login Theme                 --
Client Protocol             openid-connect
Access Type                 Public
Standard Flow Enabled           ON
Implicit Flow Enabled           OFF
Direct Access Grants Enabled        OFF
Authorization Enabled           OFF
Root URL                https://testsite.com/
* Valid Redirect URIs           https://testsite.com/* http://testsite.com/*
 Base URL               https://testsite.com/
Admin URL               --
Web Origins                 +

My keycloak config on wildfly app side:

{
  "realm": "realm_name",
  "auth-server-url": "https://testsite.com/auth",
  "ssl-required": "external",
  "resource": "client-web",
  "public-client": true,
  "use-resource-role-mappings": true,
  "confidential-port": 0
}

Expected result: authenticating without errors when service is scaled to more than 1 containers.

Answer 1

I'm using Kubernetes, and I was able to solve a problem that occurred with redirects failing when the number of replicas exceeded 1 by setting KC_CACHE_STACK to kubernetes.

  - name: KC_CACHE_STACK
    value: "kubernetes"

Answer 2

Finally, we got it working, the redirects issue appears when keycloak accepts your authentication and sends you back to your back-end application, if you have more than one replica of your back-end, the swarm balancer sends you to another instance of your back-end application instead of the one you started with, but as this instance is not authenticated yet it redirects you to keycloak again in a loop.

In our case, we just got rid of the nginx and started using the traefik docker image, this image makes sure your services connections are sticky. This would be the traefik service:

loadbalancer:
   image: traefik:1.7
   command: --docker \
     --docker.swarmmode \
     --docker.watch \
     --web \
     --loglevel=DEBUG
   ports:
     - 80:80
     - 9090:8080
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
   deploy:
     restart_policy:
       condition: any
     mode: replicated
     replicas: 1
     update_config:
       delay: 2s

And after this, you need to pass this labels to the services that are interacting (keycloak and your back-end) in the docker-compose file:

   labels:
     - "traefik.docker.network=your_network_name"
     - "traefik.port=your_service_port"
     - "traefik.frontend.rule=PathPrefix:/your_service_path;" // may be not necessary (play around)
     - "traefik.backend.loadbalancer.stickiness=true"

CAUTION!!! Passing docker.socket directly to your traefik service is considered to be a security bad practice, there are other ways to implement this more securely (https://github.com/containous/traefik/issues/4174).