Add Sentry Log4j2 appender at runtime

Question

I've been browsing previous threads about adding Log4j2 appenders at runtime but none of them really seem to fit my scenario.

We have a longrunning Flink Job packaged into a FAT jar that we essentially submit to a running Flink cluster. We want to forward error logs to Sentry. Conveniently enough Sentry provides a Log4j2 appender that I want to be able to use, but all attempts to get Log4j2 to work have failed -- going a bit crazy about this (spent days).

Since Flink (who also uses log4j2) provides a set of default logging configurations that takes precedence of any configuration files we bundle in our jar; I'm essentially left with attempting to configure the appender at runtime to see if that will make it register the appender and forward the LogEvents to it.

As a side note: I attempted to override the Flink provided configuration file (to essentially add the appender directly into the Log4j2.properties file) but Flink fails to load the plugin due to a missing dependency - io.sentry.IHub - which doesn't make sense since all examples/sentry docs don't mention any other dependencies outside of log4j related ones which already exists in the classpath.

I've followed the example in the log4j docs: Programmatically Modifying the Current Configuration after Initialization but the logs are not getting through to Sentry.

SentryLog4j.scala

package com.REDACTED.thoros.config

import io.sentry.log4j2.SentryAppender
import org.apache.logging.log4j.Level
import org.apache.logging.log4j.LogManager
import org.apache.logging.log4j.core.LoggerContext
import org.apache.logging.log4j.core.config.AppenderRef
import org.apache.logging.log4j.core.config.Configuration
import org.apache.logging.log4j.core.config.LoggerConfig

object SentryLog4j2 {
 val SENTRY_LOGGER_NAME = "Sentry"
 val SENTRY_BREADCRUMBS_LEVEL: Level = Level.ALL
 val SENTRY_MINIMUM_EVENT_LEVEL: Level = Level.ERROR
 val SENTRY_DSN =
   "REDACTED"

 def init(): Unit = {
   // scalafix:off
   val loggerContext: LoggerContext =
     LogManager.getContext(false).asInstanceOf[LoggerContext]
   val configuration: Configuration = loggerContext.getConfiguration

   val sentryAppender: SentryAppender = SentryAppender.createAppender(
     SENTRY_LOGGER_NAME,
     SENTRY_BREADCRUMBS_LEVEL,
     SENTRY_MINIMUM_EVENT_LEVEL,
     SENTRY_DSN,
     false,
     null
   )
   sentryAppender.start()
   configuration.addAppender(sentryAppender)
   // Creating a new dedicated logger for Sentry
   val ref: AppenderRef =
     AppenderRef.createAppenderRef("Sentry", null, null)
   val refs: Array[AppenderRef] = Array(ref)
   val loggerConfig: LoggerConfig = LoggerConfig.createLogger(
     false,
     Level.ERROR,
     "org.apache.logging.log4j",
     "true",
     refs,
     null,
     configuration,
     null
   )
   loggerConfig.addAppender(sentryAppender, null, null)
   configuration.addLogger("org.apache.logging.log4j", loggerConfig)
   println(configuration.getAppenders)
   loggerContext.updateLoggers()
   // scalafix:on
 }
}

Then invoking the SentryLog4j.init() in the Main module.

import org.apache.logging.log4j.LogManager
import org.apache.logging.log4j.Logger
import org.apache.logging.log4j.core.LoggerContext
import org.apache.logging.log4j.core.config.Configuration

object Main {
  val logger: Logger = LogManager.getLogger()
  sys.env.get("ENVIRONMENT") match {
    case Some("dev") | Some("staging") | Some("production") =>
      SentryLog4j2.init()
    case _ => SentryLog4j2.init() // <-- this was only added during debugging
  }

    def main(args: Array[String]): Unit = {
       logger.error("test") // this does not forward the logevent to the appender
    }
}

I think I somehow need to register the appender to loggerConfig that the rootLogger uses so that all logger.error statements are propogated to the configured Sentry appender?

Greatly appreciate any guidance with this!

Answer 1

Although not an answer to how you get log4j2 and the SentryAppender to work. For anyone else that is stumbling on this problem, I'll just briefly explain what I did to get the sentry integration working. What I eventually decided to do was drop the use of the SentryAppender and instead used the raw sentry client. Adding a wrapper class exposing the typical debug, info, warn and error methods. Then for the warn+ methods, I'd also send the logevent to Sentry.

This is essentially the only way I got this to work within a Flink cluster.

See example below:

sealed trait LoggerLike {
  type LoggerFn = (String, Option[Object]) => Unit
  val debug: LoggerFn
  val info: LoggerFn
  val warn: LoggerFn
  val error: LoggerFn
}


trait LazyLogging {
  @transient
  protected lazy val logger: CustomLogger =
    CustomLogger.getLogger(getClass.getName, enableSentry = true)
}


final class CustomLogger(slf4JLogger: Logger) extends LoggerLike {...your implementation...}

Then for each class/object (scala language at least), you'd just extend the LazyLogging trait to get a logger instance.