Logback messages getting funneled to a single logger

Question

I am facing a problem with Apache NiFi logging. NiFi is written in Java and uses logback.

It appears that ALL log messages are somehow getting funneled to the org.apache.nifi.StdOut logger. Below is an example log message that was written to org.apache.nifi.StdOut, but should have gone to the logger named org.eclipse.jetty.annotations.AnnotationParser:

2022-09-21 14:21:17,316 INFO [NiFi logging handler] org.apache.nifi.StdOut 14:21:17.316 [NiFi Web Server-22] DEBUG org.eclipse.jetty.annotations.AnnotationParser - Scanning class from jar file:///opt/nifi/work/jetty/nifi-standard-content-viewer-1.17.0.war/webapp/WEB-INF/lib/avro-1.11.0.jar!/org/apache/avro/SchemaBuilder$FieldBuilder.class->org/apache/avro/SchemaBuilder$FieldBuilder.class[0]

Why are all logback log messages getting sent to org.apache.nifi.StdOut? Why is logback not sending log messages to the approprate logger?

For reference, I am running NiFi inside of a Docker container. (Maybe that is affecting standard out?)

Here is the logback.xml configuration in use:

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
    <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" />

    <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
        <resetJUL>true</resetJUL>
    </contextListener>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="org.apache.nifi" level="INFO"/>
    <logger name="org.apache.nifi.processors" level="WARN"/>
    <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
    <logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
    <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" />

    <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
    <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
    <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" />
    <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" level="ERROR" />
    <logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
    <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
    <logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" />
    <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" level="ERROR" />

    <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />

    <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" />
    <logger name="org.apache.curator.ConnectionState" level="OFF" />

    <!-- Logger for managing logging statements for nifi clusters. -->
    <logger name="org.apache.nifi.cluster" level="INFO"/>

    <!-- Logger for logging HTTP requests received by the web server. -->
    <logger name="org.apache.nifi.server.JettyServer" level="INFO"/>

    <!-- Logger for managing logging statements for jetty -->
    <logger name="org.eclipse.jetty" level="INFO"/>

    <!-- Suppress non-error messages due to excessive logging by class or library -->
    <logger name="org.springframework" level="ERROR"/>

    <!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) -->
    <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>

    <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a large amount of WARNS. Issue described in NIFI-5479. -->
    <logger name="org.eclipse.jetty.annotations.AnnotationParser" level="ERROR"/>

    <!-- Suppress non-error messages from SSHJ which was emitting large amounts of INFO logs by default -->
    <logger name="net.schmizz.sshj" level="WARN" />
    <logger name="com.hierynomus.sshj" level="WARN" />

    <!-- Suppress non-error messages from SMBJ which was emitting large amounts of INFO logs by default -->
    <logger name="com.hierynomus.smbj" level="WARN" />

    <!-- Suppress non-error messages from AWS KCL which was emitting large amounts of INFO logs by default -->
    <logger name="com.amazonaws.services.kinesis" level="WARN" />

    <!-- Suppress non-error messages from Apache Atlas which was emitting large amounts of INFO logs by default -->
    <logger name="org.apache.atlas" level="WARN" />

    <logger name="org.apache.nifi.web.security.requests" level="INFO" />

    <!--
        Logger for capturing user events. We do not want to propagate these
        log events to the root logger. These messages are only sent to the
        user-log appender.
    -->
    <logger name="org.apache.nifi.web.security" level="INFO" />
    <logger name="org.apache.nifi.web.api.config" level="INFO" />
    <logger name="org.apache.nifi.authorization" level="INFO" />
    <logger name="org.apache.nifi.cluster.authorization" level="INFO" />
    <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" />
    <logger name="org.springframework.security.saml.log" level="WARN" />
    <logger name="org.opensaml" level="WARN" />
    <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" />
    <logger name="org.apache.nifi.bootstrap" level="INFO" />
    <logger name="org.apache.nifi.bootstrap.Command" level="INFO" />

    <logger name="org.apache.nifi.StdOut" level="INFO" />
    <logger name="org.apache.nifi.StdErr" level="INFO" />

    <root level="INFO">
        <appender-ref ref="CONSOLE" />
    </root>

</configuration>

Answer 1

The configuration is not setup well. In this case, you can setup as:

<logger name="org.eclipse.jetty.annotations.AnnotationParser" level="ERROR"/ **additivity=false**>
  <appender-ref ref="CONSOLE" />
</logger>

You may want to specified appender other than console appender, find more appender here: file appender

Why log pattern shows up twice, and why the message is sent to root logger but not the specified logger? They are actually same question. In your logger configuration, you missed appender, thus the only appender available is root logger.

By default log message are sent to specified its own logger/appender and parent logger/appender, if you want log send to only specified logger, you can set additivity=false in logger config. logger config has additivity = true if it's not specified.

logback logger additivity

Answer 2

it seems to me that the StdOut Logger is outputting the format of all your loggers within the message they log, why dont you try forcing a different appender only for StdOut and StdErr?

In your last loggers, change the appender:

<logger name="org.apache.nifi.StdOut" level="INFO" additivity="false">
    <appender-ref ref="PLAIN" />
</logger>

<logger name="org.apache.nifi.StdErr" level="INFO" additivity="false">
    <appender-ref ref="PLAIN" />
</logger>

And adding the dummy pattern appender:

<appender name="PLAIN">
    <encoder>
        <pattern>%msg%n</pattern>
    </encoder>
</appender>

I believe that should do the trick