Store app logs in AWS Cloudwatch using Log4J2 appender

Question

I'm trying to store application-level logs in AWS Cloudwatch logs from spark structured streaming application

I am trying to using Log4J2 cloudwatch appender from following github repo (link is https://github.com/kdgregory/log4j-aws-appenders/tree/trunk)

Program is working and successfully loading streaming logs into Cloudwatch when i am running from intellij idea. However when i am trying to convert it as jar file and try to run from ec2 instance, its throwing following error:

2024-12-03T07:06:05.291599Z main ERROR Error processing element CloudWatchAppender ([Appenders: null]): CLASS_NOT_FOUND

Log4j2 properties file:

status = debug
name= LoggingConfig

# ConsoleAppender will print logs on console
appender.console.type = Console
appender.console.name = consoleLogger
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
 
 # Specify the pattern of the logs
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%c] - %msg%n
 
#Specify the pattern of the cloudwatch logs
appender.cloudwatch= com.kdgregory.log4j2.aws.CloudWatchAppender
appender.cloudwatch.type= CloudWatchAppender
appender.cloudwatch.name = CLOUDWATCH
appender.cloudwatch.layout.type = PatternLayout
appender.cloudwatch.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%c] - %msg%n
appender.cloudwatch.logGroup= sparklog
appender.cloudwatch.logStream= cloudwatch-{date}-{hostname}-${awslogs:pid}


# Mention package name here in place of example. Classes in this package or subpackages will use ConsoleAppender and CLOUDWATCH for logging
logger.app.name = sparkapp
logger.app.level = info
logger.app.additivity = false
logger.app.appenderRef.console.ref = consoleLogger
logger.app.appenderRef.cloudwatch.ref = CLOUDWATCH
 
 
#Configure root logger for logging error logs in classes which are in package other than above specified package
rootLogger.level = info
rootLogger.additivity = false
rootLogger.appenderRef.console.ref = consoleLogger
rootLogger.appenderRef.console.level = error
rootLogger.appenderRef.cloudwatch.ref = CLOUDWATCH
 

Following are the dependencies which i have used:

dependencies {

//Logging
implementation 'org.apache.logging.log4j:log4j-core:2.23.1'
implementation 'org.apache.logging.log4j:log4j-api:2.23.1'

// Dependencies for aws cloudwatch log
implementation 'com.amazonaws:aws-java-sdk-logs:1.12.765'
implementation 'software.amazon.awssdk:cloudwatchlogs:2.26.30'

// Dependencies for log4j2 appender for aws cloud watch
implementation 'com.kdgregory.logging:log4j2-aws-appenders:3.2.1'
implementation 'com.kdgregory.logging:aws-facade-v2:3.2.1'
implementation 'com.kdgregory.logging:logwriters:3.2.1'

}

Please Note No issues when running application from ide. Only issue when submitting application as jar file.

Following is the error while running application using application fat jar or thin jar along with all jar dependencies:

2024-12-03 07:23:07,843 main ERROR Unable to locate plugin type for CloudWatchAppender 2024-12-03 07:23:07,890 main WARN Level is ignored when levelAndRefs syntax is used. 2024-12-03 07:23:07,891 main WARN Appender references are ignored when levelAndRefs syntax is used 2024-12-03 07:23:07,916 main ERROR Unable to locate plugin for CloudWatchAppender 2024-12-03 07:23:07,917 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.config.AppendersPlugin for element Appenders: java.lang.NullPointerException java.lang.NullPointerException at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:52) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.generateParameters(PluginBuilder.java:264) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:137) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1122) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1047) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:651) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:249) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:295) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:621) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:245) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:176) at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:666) at org.apache.logging.log4j.LogManager.getRootLogger(LogManager.java:700) at org.apache.spark.internal.Logging.initializeLogging(Logging.scala:130) at org.apache.spark.internal.Logging.initializeLogIfNecessary(Logging.scala:115) at org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:109) at org.apache.spark.deploy.SparkSubmit.initializeLogIfNecessary(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:83) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

2024-12-03 07:23:07,918 main ERROR Unable to locate appender "STDOUT" for logger config "root" 2024-12-03 07:23:07,919 main ERROR Unable to locate appender "cloudwatch" for logger config "root" 2024-12-03 07:23:07,919 main ERROR Unable to locate appender "consoleLogger" for logger config "sparkapp" 2024-12-03 07:23:07,920 main ERROR Unable to locate appender "CLOUDWATCH" for logger config "sparkapp"

Answer 1

Hi Instead of log4j2 appender, i have installed aws cloudwatch agent on ec2 instance and pushed spark application logs on ec2 instance to Cloudwatch.