CODEWITHSUNDEEP
javalog4j2
Evel Knievel
Evel Knievel

Reputation: 255

How to suppress duplicate messages in log4j2

I am quite new to Java and log4j2, so sorry for the maybe odd question. My problem is the following. I wrote an application which uses log4j2 for logging. The program analyses data and writes a warning in the case the given string could not be parsed as wanted. Sometimes the program gets a lot of unexpected strings and thus it logs all time the same error message. So, the question is, how to avoid to log the same error message over and over again. Instead for instance to see the same error message 2000 times in the logfile, I would like to have a hint in the logfile that this error message was written x-times. My current log4j2 config file look as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Properties>
        <Property name="pattern">%d{DEFAULT} %-5p %-18.18c %4.4L [%-15.15t] %m%n</Property>
    </Properties>

    <Appenders>
        <Console name="STDERR" target="SYSTEM_ERR">
            <PatternLayout pattern="${pattern}" />
        </Console>
    </Appenders>
    <Loggers>
        <Root level="warn">
            <AppenderRef ref="STDERR" />
        </Root>
    </Loggers>
</Configuration>

Upvotes: 6

Views: 3171

Answers (2)

Remko Popma
Remko Popma

Reputation: 36794

It is of course possible to create a log4j2 plugin that does this but I would argue that the logging library is not the best place to address this.

I think you will end up with much simpler code if in your application you keep track of the previous message you logged. If the next message will be the same then don't log but increment a counter instead. When you hit a different message you log the counter value.

(This can be made more general with a Dequeue to track multiple messages.)

Upvotes: 0

pandaadb
pandaadb

Reputation: 6456

I sadly don't have the code on me currently, but this is how you can solve this (and there might very well be a solution out there already).

You can create a new appender that logs asynchronously to your file/console whatever.

In the appender, you can implement aggregation yourself. Whenever an logevent comes in, instead of writing it out immediately, check if you've seen the message before. If you did, instead of logging it, increment your counter.

Eventually, transform those aggregated log events to an aggregated message and process that one instead of your thousand log events.

This could be solved (for example) with a second worker on your asynchronous appender base. Usually (I am looking at lockback, but I assume it will be fairly similar for you), the asynchronous appenders end up adding events to a queue which will be processed by a different thread that works off that queue.

You can have a different queue for your aggregated messages, and a worker that runs say every 5 seconds. If in those 5 seconds the same message has been seen multiple times, your worker can aggregate that message and put it on the queue to be processed so it eventually ends up in the logs.

I hope that helps,

Artur

P.S.: With regards to your config, simply replace the Console appender with your custom aggregating appender. If you want console, simply extend that appender and make it do what you want it to do :)

Upvotes: 0

Related Questions