OpenShift: How to set the log-level of JBoss AS

Question

In OpenShift I want to set the log level of JBoss AS to "DEBUG".

In my local JBoss installation I can achieve this by editing the logging configuration in JBOSS_HOME/standalone/configuration/standalone.xml.

The default level of the root-logger is "INFO", but when I set that to "DEBUG" and restart JBoss, then I have achieved my goal.

How can I acheve the same in OpenShift? I can SSH into the OpenShift App and manually modify jbossas/standalone/configuration/standalone.xml. However when I restart the App, all those changes are lost, and consequently the level of the root-logger is reset to "INFO".

Answer 1

Create a custom script(bash/sh, whichever is available), copy them into /opt/data/deployment/extensionpoint of your image with help of jib by configuring extraDirectories, so the script will run when your server starts.

The script should wait for your application to start by looping the healthcheck(for example, by curl some endpoint and expect http 200 periodically, if curl is available in your base image); when it's up, start to call jboss-cli.sh --file=<some-file-containing-operations-you-want>.

This file could be copied into image using jib also, or, let your script build it in runtime, and put it into some dir your app user could use, for example /tmp.

The file should contain commands of

1. try to query the logger existence(/subsystem=logging/logger=my.package:read-resource), if output==failed, then the logger does not exist, you must add it first;

2. /subsystem=logging/logger=my.package:write-attribute(name="level", value="WARN"), for example.

See: https://docs.redhat.com/en/documentation/red_hat_jboss_enterprise_application_platform/7.2/html/management_cli_guide/batch_processing#batch_processing

and

https://docs.redhat.com/en/documentation/red_hat_jboss_enterprise_application_platform/6.4/html/administration_and_configuration_guide/sect-logging_configuration_in_the_cli#Configure_a_Console_Log_Handler_in_the_CLI

Doc version differ, but the basic part regarding this topic should be the same.

If your base image is distroless, then this approach does not work, as it does not allow custom scripts to run in shell.

Answer 2

The short version:

1. Create a config map that contains the file you want to mount in the container (e.g. logging.properties or standalone.xml or any file)
2. Update the JBOSS deployment to mount the config map in the container

The long version:

Here are my personal notes - https://www.freekb.net/Article?id=5859

Let's say logging.properties file in the JBOSS pod contains something like this and you want to update logger.level to DEBUG.

~]$ oc exec pod/eap74-openjdk8-openshift-rhel7-8d49d6bd7-j6blz -- cat 
/opt/eap/standalone/configuration/logging.properties
# Note this file has been generated and will be overwritten if a
# logging subsystem has been defined in the XML configuration.


# Additional loggers to configure (the root logger is always configured)
logger.level=INFO
logger.handlers=CONSOLE
logger.sun.rmi.level=WARN
logger.sun.rmi.useParentHandlers=true
logger.org.jboss.as.config.level=DEBUG
logger.org.jboss.as.config.useParentHandlers=true
logger.io.jaegertracing.Configuration.level=WARN
logger.io.jaegertracing.Configuration.useParentHandlers=true
logger.com.arjuna.level=WARN
logger.com.arjuna.useParentHandlers=true

handler.CONSOLE=org.jboss.logmanager.handlers.ConsoleHandler
handler.CONSOLE.level=ALL
handler.CONSOLE.formatter=COLOR-PATTERN
handler.CONSOLE.properties=autoFlush,target,enabled
handler.CONSOLE.autoFlush=true
handler.CONSOLE.target=SYSTEM_OUT
handler.CONSOLE.enabled=true

# Additional formatters to configure
formatters=OPENSHIFT
formatter.COLOR-PATTERN=org.jboss.logmanager.formatters.PatternFormatter
formatter.COLOR-PATTERN.properties=pattern
formatter.OPENSHIFT=org.jboss.logmanager.formatters.JsonFormatter
formatter.OPENSHIFT.constructorProperties=keyOverrides
formatter.OPENSHIFT.keyOverrides=timestamp\=@timestamp
formatter.OPENSHIFT.exceptionOutputType=FORMATTED
formatter.OPENSHIFT.metaData=@version\=1
formatter.OPENSHIFT.prettyPrint=false
formatter.OPENSHIFT.printDetails=false
formatter.OPENSHIFT.recordDelimiter=\n

Let's start by using the oc cp (copy) command to copy the logging.properties file in the pod to your host system.

oc cp eap74-openjdk8-openshift-rhel7-8d49d6bd7-j6blz:/opt/eap/standalone/configuration/logging.properties /tmp/logging.properties

Let's make some change to the /tmp/logging.properties file, such as changing logger.level=INFO to logger.level=DEBUG. The oc create configmap command can be used to create a config map that contains the content of the logging.properties file.

oc create configmap logging-properties --from-file /tmp/logging.properties

The oc get deployments command can be used to list the deployments.

~]$ oc get deployments
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
eap74-openjdk8-openshift-rhel7   1/1     1            1           8d

The oc edit deployment can be used to update the deployment to mount the configmap to /opt/eap/standalone/configuration/logging.properties in the container.

apiVersion: v1
items:
  spec:
    template:
      spec:
        containers:
        - image: registry.redhat.io/jboss-eap-7/eap74-openjdk8-openshift-rhel7@sha256:02160a5f66638faa74d44e7332925c3210d986e74bf256dd17d4876715e05ff9
          imagePullPolicy: IfNotPresent
          name: eap74-openjdk8-openshift-rhel7
          ports:
          - containerPort: 8080
            protocol: TCP
          - containerPort: 8443
            protocol: TCP
          - containerPort: 8778
            protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/eap/standalone/configuration/logging.properties
            name: my-logging-properties
            subPath: logging.properties
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 30
        volumes:
        - configMap:
            defaultMode: 420
            name: logging-properties
          name: my-logging-properties

Once the deployment has been updated a new pod should immediate get spawned. The oc get pods command can be used to list the pods.

~]$ oc get pods
NAME                                             READY   STATUS    RESTARTS   AGE
eap74-openjdk8-openshift-rhel7-8d49d6bd7-j6blz   1/1     Running   0          9m42s

The oc exec command can be used to verify that the logging.properties file in the pod contains whatever change you made to the file, such as having logger.level=DEBUG in this example.

~]$ oc exec pod/eap74-openjdk8-openshift-rhel7-8d49d6bd7-j6blz -- cat /opt/eap/standalone/configuration/logging.properties
# Note this file has been generated and will be overwritten if a
# logging subsystem has been defined in the XML configuration.


# Additional loggers to configure (the root logger is always configured)
logger.level=DEBUG
logger.handlers=CONSOLE
logger.sun.rmi.level=WARN
logger.sun.rmi.useParentHandlers=true
logger.org.jboss.as.config.level=DEBUG
logger.org.jboss.as.config.useParentHandlers=true
logger.io.jaegertracing.Configuration.level=WARN
logger.io.jaegertracing.Configuration.useParentHandlers=true
logger.com.arjuna.level=WARN
logger.com.arjuna.useParentHandlers=true

handler.CONSOLE=org.jboss.logmanager.handlers.ConsoleHandler
handler.CONSOLE.level=ALL
handler.CONSOLE.formatter=COLOR-PATTERN
handler.CONSOLE.properties=autoFlush,target,enabled
handler.CONSOLE.autoFlush=true
handler.CONSOLE.target=SYSTEM_OUT
handler.CONSOLE.enabled=true

# Additional formatters to configure
formatters=OPENSHIFT
formatter.COLOR-PATTERN=org.jboss.logmanager.formatters.PatternFormatter
formatter.COLOR-PATTERN.properties=pattern
formatter.OPENSHIFT=org.jboss.logmanager.formatters.JsonFormatter
formatter.OPENSHIFT.constructorProperties=keyOverrides
formatter.OPENSHIFT.keyOverrides=timestamp\=@timestamp
formatter.OPENSHIFT.exceptionOutputType=FORMATTED
formatter.OPENSHIFT.metaData=@version\=1
formatter.OPENSHIFT.prettyPrint=false
formatter.OPENSHIFT.printDetails=false
formatter.OPENSHIFT.recordDelimiter=\n

Answer 3

If you want this settings to be permanent you should add something to *.openshift/action_hooks/deploy* which applies this setting during deploy of the application. The hook is part of the git repo, so you can modify it locally and it gets executed during deployment. See also https://www.openshift.com/developers/deploying-and-building-applications.

You could use a perl one lines to do the modification. Something like this should do:

perl -p -i -e 's/<level name=\"INFO\"\/>/<level name=\"DEBUG\"\/>/g' <path-tp-standalone-xml>