CODEWITHSUNDEEP
javajbosswildflykeycloak
user14140381
user14140381

Reputation: 251

Keycloak - cannot start embedded server: WFLYEMB0022: Cannot invoke 'start' on embedded process: Parameter 'abstractPath' must not be empty

I have a Keycloak instance running standalone on a Heroku PM dyno via the container registry. Recently, this app has crashed with the logs below. I have not changed the code since May 25th at which point everything was working. The heroku dyno is not near any limits. My local version is fine. Therefore, I think one of the following must have happened.

Debugging I have tried:

What should I try next?

Logs:

Jun 09 08:50:28 Release v66 created by user [EMAIL]
Jun 09 08:50:28 Deployed web (c19fbf3aed30) by user [EMAIL]
Jun 09 08:50:39 Starting process with command `-b 0.0.0.0`
Jun 09 08:50:40  Found database configuration in [CORRECT DETAILS WERE HERE]
Jun 09 08:50:43  Added [EMAIL] to '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user
Jun 09 08:50:43  -b 0.0.0.0
Jun 09 08:50:43  =========================================================================
Jun 09 08:50:43    Using PostgreSQL database
Jun 09 08:50:43  =========================================================================
Jun 09 08:50:44  15:50:44,394 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.2.Final
Jun 09 08:50:44  15:50:44,451 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.12.Final
Jun 09 08:50:44  15:50:44,459 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
Jun 09 08:50:44  15:50:44,576 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 12.0.0 (WildFly Core 13.0.3.Final) starting
Jun 09 08:50:44  15:50:44,611 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.as: org.jboss.msc.service.StartException in service jboss.as: Failed to start service
Jun 09 08:50:44     at [email protected]//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
Jun 09 08:50:44     at [email protected]//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
Jun 09 08:50:44     at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
Jun 09 08:50:44     at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
Jun 09 08:50:44     at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
Jun 09 08:50:44     at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
Jun 09 08:50:44     at java.base/java.lang.Thread.run(Thread.java:834)
Jun 09 08:50:44  Caused by: java.lang.IllegalArgumentException: COM00008: Parameter 'abstractPath' must not be empty
Jun 09 08:50:44     at [email protected]//org.wildfly.common.Assert.checkNotEmptyParam(Assert.java:104)
Jun 09 08:50:44     at [email protected]//org.jboss.as.controller.services.path.AbsolutePathService.convertPath(AbsolutePathService.java:70)
Jun 09 08:50:44     at [email protected]//org.jboss.as.controller.services.path.AbsolutePathService.<init>(AbsolutePathService.java:49)
Jun 09 08:50:44     at [email protected]//org.jboss.as.controller.services.path.AbsolutePathService.addService(AbsolutePathService.java:59)
Jun 09 08:50:44     at [email protected]//org.jboss.as.controller.services.path.AbsolutePathService.addService(AbsolutePathService.java:53)
Jun 09 08:50:44     at [email protected]//org.jboss.as.controller.services.path.PathManagerService.addAbsolutePathService(PathManagerService.java:259)
Jun 09 08:50:44     at [email protected]//org.jboss.as.controller.services.path.PathManagerService.addHardcodedAbsolutePath(PathManagerService.java:160)
Jun 09 08:50:44     at [email protected]//org.jboss.as.server.ServerPathManagerService.addService(ServerPathManagerService.java:55)
Jun 09 08:50:44     at [email protected]//org.jboss.as.server.ApplicationServerService.start(ApplicationServerService.java:179)
Jun 09 08:50:44     at [email protected]//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
Jun 09 08:50:44     at [email protected]//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
Jun 09 08:50:44     ... 6 more
Jun 09 08:50:44  Cannot start embedded server: WFLYEMB0022: Cannot invoke 'start' on embedded process: WFLYSRV0141: Cannot start server: JBTHR00005: Operation failed: Failed to start service: COM00008: Parameter 'abstractPath' must not be empty

My docker entry point file

 
# Set database config from Heroku DATABASE_URL or HEROKU_POSTGRESQL_ORANGE_URL
if [ "$DATABASE_URL" != "" ]; then
   echo "Found database configuration in DATABASE_URL=$DATABASE_URL"
 
   regex='^postgres://([a-zA-Z0-9_-]+):([a-zA-Z0-9]+)@([a-z0-9.-]+):([[:digit:]]+)/([a-zA-Z0-9_-]+)$'
   if [[ $DATABASE_URL =~ $regex ]]; then
       export DB_ADDR=${BASH_REMATCH[3]}
       export DB_PORT=${BASH_REMATCH[4]}
       export DB_DATABASE=${BASH_REMATCH[5]}
       export DB_USER=${BASH_REMATCH[1]}
       export DB_PASSWORD=${BASH_REMATCH[2]}
 
       echo "DB_ADDR=$DB_ADDR, DB_PORT=$DB_PORT, DB_DATABASE=$DB_DATABASE, DB_USER=$DB_USER, DB_PASSWORD=$DB_PASSWORD"
       export DB_VENDOR=postgres
   fi
 
fi
 
# usage: file_env VAR [DEFAULT]
#    ie: file_env 'XYZ_DB_PASSWORD' 'example'
# (will allow for "$XYZ_DB_PASSWORD_FILE" to fill in the value of
#  "$XYZ_DB_PASSWORD" from a file, especially for Docker's secrets feature)
file_env() {
   local var="$1"
   local fileVar="${var}_FILE"
   local def="${2:-}"
   if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
       echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
       exit 1
   fi
   local val="$def"
   if [ "${!var:-}" ]; then
       val="${!var}"
   elif [ "${!fileVar:-}" ]; then
       val="$(< "${!fileVar}")"
   fi
   export "$var"="$val"
   unset "$fileVar"
}
 
##################
# Add admin user #
##################
 
file_env 'KEYCLOAK_USER'
file_env 'KEYCLOAK_PASSWORD'
 
if [ $KEYCLOAK_USER ] && [ $KEYCLOAK_PASSWORD ]; then
   /opt/jboss/keycloak/bin/add-user-keycloak.sh --user $KEYCLOAK_USER --password $KEYCLOAK_PASSWORD || echo "User already exists."
fi
 
############
# Hostname #
############
 
if [ "$KEYCLOAK_HOSTNAME" != "" ]; then
   SYS_PROPS="-Dkeycloak.hostname.provider=fixed -Dkeycloak.hostname.fixed.hostname=$KEYCLOAK_HOSTNAME"
 
   if [ "$KEYCLOAK_HTTP_PORT" != "" ]; then
       SYS_PROPS+=" -Dkeycloak.hostname.fixed.httpPort=$KEYCLOAK_HTTP_PORT"
   fi
 
   if [ "$KEYCLOAK_HTTPS_PORT" != "" ]; then
       SYS_PROPS+=" -Dkeycloak.hostname.fixed.httpsPort=$KEYCLOAK_HTTPS_PORT"
   fi
fi
 
################
# Realm import #
################
 
if [ "$KEYCLOAK_IMPORT" ]; then
   SYS_PROPS+=" -Dkeycloak.import=$KEYCLOAK_IMPORT"
fi
 
########################
# JGroups bind options #
########################
 
if [ -z "$BIND" ]; then
   BIND=$(hostname -i)
fi
if [ -z "$BIND_OPTS" ]; then
   for BIND_IP in $BIND
   do
       BIND_OPTS+=" -Djboss.bind.address=$BIND_IP -Djboss.bind.address.private=$BIND_IP "
   done
fi
SYS_PROPS+=" $BIND_OPTS"
 
#################
# Configuration #
#################
 
# If the server configuration parameter is not present, append the HA profile.
if echo "$@" | egrep -v -- '-c |-c=|--server-config |--server-config='; then
   SYS_PROPS+=" -c=standalone-ha.xml"
fi
 
############
# DB setup #
############
 
file_env 'DB_USER'
file_env 'DB_PASSWORD'
 
# Lower case DB_VENDOR
DB_VENDOR=`echo $DB_VENDOR | tr A-Z a-z`
 
# Detect DB vendor from default host names
if [ "$DB_VENDOR" == "" ]; then
   if (getent hosts postgres &>/dev/null); then
       export DB_VENDOR="postgres"
   elif (getent hosts mysql &>/dev/null); then
       export DB_VENDOR="mysql"
   elif (getent hosts mariadb &>/dev/null); then
       export DB_VENDOR="mariadb"
   fi
fi
 
# Detect DB vendor from legacy `*_ADDR` environment variables
if [ "$DB_VENDOR" == "" ]; then
   if (printenv | grep '^POSTGRES_ADDR=' &>/dev/null); then
       export DB_VENDOR="postgres"
   elif (printenv | grep '^MYSQL_ADDR=' &>/dev/null); then
       export DB_VENDOR="mysql"
   elif (printenv | grep '^MARIADB_ADDR=' &>/dev/null); then
       export DB_VENDOR="mariadb"
   fi
fi
 
# Default to H2 if DB type not detected
if [ "$DB_VENDOR" == "" ]; then
   export DB_VENDOR="h2"
fi
 
# Set DB name
case "$DB_VENDOR" in
   postgres)
       DB_NAME="PostgreSQL";;
   mysql)
       DB_NAME="MySQL";;
   mariadb)
       DB_NAME="MariaDB";;
   h2)
       DB_NAME="Embedded H2";;
   *)
       echo "Unknown DB vendor $DB_VENDOR"
       exit 1
esac
 
# Append '?' in the beggining of the string if JDBC_PARAMS value isn't empty
export JDBC_PARAMS=$(echo ${JDBC_PARAMS} | sed '/^$/! s/^/?/')
 
# Convert deprecated DB specific variables
function set_legacy_vars() {
 local suffixes=(ADDR DATABASE USER PASSWORD PORT)
 for suffix in "${suffixes[@]}"; do
   local varname="$1_$suffix"
   if [ ${!varname} ]; then
     echo WARNING: $varname variable name is DEPRECATED replace with DB_$suffix
     export DB_$suffix=${!varname}
   fi
 done
}
set_legacy_vars `echo $DB_VENDOR | tr a-z A-Z`
 
# Configure DB
 
echo "========================================================================="
echo ""
echo "  Using $DB_NAME database"
echo ""
echo "========================================================================="
echo ""
 
if [ "$DB_VENDOR" != "h2" ]; then
   /bin/sh /opt/jboss/tools/databases/change-database.sh $DB_VENDOR
fi
 
/opt/jboss/tools/x509.sh
/opt/jboss/tools/jgroups.sh $JGROUPS_DISCOVERY_PROTOCOL $JGROUPS_DISCOVERY_PROPERTIES
/opt/jboss/tools/autorun.sh
 
##################
# Start Keycloak #
##################
 
exec /opt/jboss/keycloak/bin/standalone.sh $SYS_PROPS $@ -Djboss.http.port=$PORT
exit $?

My Dockerfile:

 
COPY docker-entrypoint.sh /opt/jboss/tools
 
ENTRYPOINT [ "/opt/jboss/tools/docker-entrypoint.sh" ]
 
CMD ["-b", "0.0.0.0"]
 
RUN mkdir -p /opt/jboss/keycloak/themes/my_theme/
COPY /my_theme/ /opt/jboss/keycloak/themes/my_theme/

Upvotes: 4

Views: 1426

Answers (1)

Chas Bean
Chas Bean

Reputation: 11

I ran into this exact same problem, and was able to resolve the issue by setting up an entirely new Keycloak Heroku app, redeploying my code there, then renaming my apps so that traffic from the old app when to the new app. I can give detailed information on this if you need, but this resulted in a fresh Keycloak server (dyno) that once scaled to Performance-M ran exactly the same as before (afaik, Keycloak configuration is stored in the underlying database, so if you attach your original database to the new dyno app then you should be fine)

I ran into the exact same problem in the past week or so. I was actually running a different version of Keycloak as well, v10, and initially attempted upgrading to v13 to resolve the problem, but that didn't do it. I think you are correct that there is some sort of corruption happening, as my instance was also untouched when the first failure occurred. I checked Heroku changelogs for anything that would affect my deployment, as the failure seemed to occur after a daily dyno restart, but I found nothing there)

I think you are correct that there is some sort of corruption or underlying bug here, likely a hard-to-reproduce one, possibly with WildFly code, JVM, Keycloak, or something related to Heroku

Upvotes: 1

Related Questions