no members discovered after 2001 ms: creating cluster as coordinator

Question

I have created a statefulsets for Keycloak. I want a highly available instance of it on AKS. I'm using Keycloak 21.1.1 It is running in production mode, with a certificate inside the pod

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: keycloak-statefulset
spec:
  serviceName: keycloak-svc
  replicas: 3
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8443"
        prometheus.io/scrape: "true"
        prometheus.io/scheme: https
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: keycloak
                topologyKey: kubernetes.io/zone
              weight: 1
      serviceAccountName: keycloak-svc-account
      containers:
      - env:
          - name: DB_HOST
            value: "10.0.194.2"
          - name: DB_PORT
            value: "5432"
          - name: DB_USERNAME
            value: keycloak
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: keycloak-postgres-secret
                key: keycloak-password
          - name: KEYCLOAK_ADMIN
            value: "admin"
          - name: KEYCLOAK_ADMIN_PASSWORD
            valueFrom:
              secretKeyRef:
                name: keycloak-secret
                key: password
          - name: KC_CACHE
            value: "ispn"
          - name: JAVA_OPTS
            value: -Djgroups.dns.query=keycloak-svc.amq-keycloak-dev.svc.cluster.local
          - name: KC_CACHE_STACK
            value: kubernetes
        name: keycloak
        image: "harbor.net/keycloak:main" // this is not the actual one
        imagePullPolicy: Always
        resources:
          requests:
            memory: "1G"
            cpu: "250m"
          limits:
            memory: "1G"
            cpu: "500m"
        ports:
          - containerPort: 8443
            name: console
            protocol: TCP
        volumeMounts:
        - name: security-keystore-config
          mountPath: /vault/keystore
        - name: security-truststore-config
          mountPath: /vault/truststore
      volumes:
        - name: security-truststore-config
          secret:
            secretName: truststore-cert-keycloak
        - name: security-keystore-config
          secret:
            secretName: keystore-cert-keycloak

I have added these env variables, and mentioned the headless service FQDN as well. The docker files looks -

SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN apt-get -qq -o=Dpkg::Use-Pty=0 update && apt-get -y install  && apt-get -y install wget
RUN apt-get -y install libaio1
RUN rm -rf /var/lib/apt/lists/*

COPY keycloak-event-metrics-1.0.1-SNAPSHOT.jar /opt/keycloak-event-metrics-1.0.1-SNAPSHOT.jar

WORKDIR /opt/keycloak
RUN wget https://github.com/keycloak/keycloak/releases/download/21.1.1/keycloak-21.1.1.tar.gz

ENV KC_DB=postgres
ENV KC_HOSTNAME_STRICT=false
ENV KC_HEALTH_ENABLED=true 
ENV KC_METRICS_ENABLED=true
ENV KC_METRICS_STATS_ENABLED=true
ENV KC_FEATURES=token-exchange
ENV KC_PROXY=passthrough
ENV KC_DB_POOL_INITIAL_SIZE=10
ENV KC_DB_POOL_MIN_SIZE=10
ENV KC_DB_POOL_MAX_SIZE=25
ENV KC_LOG_CONSOLE_COLOR=true

EXPOSE 8443

COPY docker-entrypoint.sh /assets/docker-entrypoint.sh
ENTRYPOINT ["/assets/docker-entrypoint.sh"]

#!/bin/bash
set -e

tar -xvzf /opt/keycloak/keycloak-21.1.1.tar.gz > /dev/null
rm /opt/keycloak/keycloak-21.1.1.tar.gz
cp /opt/keycloak-event-metrics-1.0.1-SNAPSHOT.jar /opt/keycloak/keycloak-21.1.1/providers/

SSL_PASS==$(date +%s | sha256sum | base64 | head -c 32 ; echo)
folder_path="/vault/truststore"
cer_files=($(find "$folder_path" -type f -name "*.cer"))

mkdir -p /vault/certificate
truststore_path="/vault/certificate/truststore.jks"

for cer_file in "${cer_files[@]}"; do
    filename=$(basename "$cer_file")
    alias_name="${filename%.*}"
    keytool -importcert -alias "$alias_name" -file "$cer_file" -keystore "$truststore_path" -storepass "$SSL_PASS" -noprompt > /dev/null 2>&1
done

folder_path="/vault/keystore/"
cer_file=($(find "$folder_path" -type f -name "*.cer"))
key_file=($(find "$folder_path" -type f -name "*.key"))
openssl pkcs12 -export -out /vault/certificate/keystore.p12 -inkey "$key_file" -in "$cer_file" -passout pass:"$SSL_PASS" > /dev/null 2>&1

keytool -importkeystore -srckeystore /vault/certificate/keystore.p12 -srcstoretype PKCS12 -destkeystore /vault/certificate/server.keystore -deststoretype JKS -srcstorepass "$SSL_PASS" -deststorepass "$SSL_PASS" > /dev/null 2>&1

export KC_DB_URL="jdbc:postgresql://$DB_HOST:$DB_PORT/keycloak?useSSL=false"
export KC_SPI_TRUSTSTORE_FILE_PASSWORD=$SSL_PASS
export KC_SPI_TRUSTSTORE_FILE_FILE=/vault/certificate/truststore.jks
export KC_HTTPS_KEY_STORE_FILE=/vault/certificate/server.keystore
export KC_HTTPS_KEY_STORE_PASSWORD=$SSL_PASS
export KC_DB_USERNAME=$DB_USERNAME
export KC_DB_PASSWORD=$DB_PASSWORD


/opt/keycloak/keycloak*/bin/kc.sh build
/opt/keycloak/keycloak*/bin/kc.sh start

Issue: When I am running more than a pod, all pods run as coordinator, i.e, The pods are not discoverable. How do I fix this?

Answer 1

Solved this. Had to make two services. One for the Keycloak to connect to ActiveMQ Artemis and other service(headless) for High Availabilty

apiVersion: v1
kind: Service
metadata:
  name: keycloak-headless-svc
spec:
  clusterIP: None
  ports:
  - port: 8443
    name: console
    protocol: TCP
    targetPort: 8443
  selector:
    app: keycloak
---
apiVersion: v1
kind: Service
metadata:
  name: keycloak-svc
spec:
  type: ClusterIP
  ports:
  - port: 8443
    name: console
    protocol: TCP
    targetPort: 8443
  selector:
    app: keycloak