Delay in Kafka Log4j appender

Question

So here is my setup, and this is all being done in Eclipse

1. I am trying to implement the Kafka Log4j appender to pump messages into Kafka.

2. I have some java code setup to create the log messages:

   public static void main(String[] args) {
   
   SimpleDateFormat sdf = new SimpleDateFormat();
   
   log.debug("Debug message at "+sdf.format(new Date()));
   
   log.info("Info message at "+sdf.format(new Date()));
   
   log.error("Error Message at "+sdf.format(new Date()));
   
   log.fatal("Fatal Message at "+sdf.format(new Date()));
   
   log.warn("Warn Message at "+sdf.format(new Date()));
   
   log.trace("Trace Message at "+sdf.format(new Date()));
   }
   

3. here are my log4j.properties

   log4j.rootLogger=TRACE, stdout
   log4j.appender.stdout=org.apache.log4j.ConsoleAppender
   log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
   log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% %m%n
   
   log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
   log4j.appender.KAFKA.BrokerList=localhost:9092
   log4j.appender.KAFKA.Topic=kfkLogs
   log4j.appender.KAFKA.producer.type=async
   log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
   log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n
   
   log4j.logger.logGen.TestLog4j=TRACE, KAFKA
   

4. In another instance of Eclipse, i have my Kafka consumer code running which basically prints out any messages it receieves

Here is my problem,

When i execute my log producer, i do not always see the messages in the consumer. sometimes i see the messages and sometimes i do not. Sometimes i can execute the producer repeatedly for a few seconds and each time i see the messages, but other times, the consumer only gets them once and not for the next minute or so. then it starts receiving again. so the messages that i pushed in the meantime are lost.

Is there anyway that there can be a delay when Kafka published the messages ?

Can it have something to do with my Kafka or Zookeeper configurations ?

I am executing everything on localhost

Please advise.

Answer 1

log4j.appender.KAFKA.producer.type=sync

fixed it.