CODEWITHSUNDEEP
apache-kafkakafka-producer-api
Lahiru
Lahiru

Reputation: 699

Use kafka producer as log appender

I've implemented a log appender with kafka producer by following the basic kafka producer documentation. My config looks like below and I use the KafkaProducer and use send() to send the each log to kafka. In my case I cannot use KafkaAppender because we don't support slf4j or log4j.

How do I batch these logs and how to optimize the below configuration to handle large number of logs.

Properties props = new Properties();
    props.put("bootstrap.servers", "ip:9092");
    props.put("acks", "all");
    props.put("retries", 0);
    props.put("batch.size", 16384);
    props.put("linger.ms", 1);
    props.put("buffer.memory", 33554432);
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Upvotes: 0

Views: 339

Answers (1)

serejja
serejja

Reputation: 23841

Increase the linger.ms config. This config means "send messages after this timeout even if the batch is not yet full". In your case your producer flushes data every millisecond and that's why you don't notice any "batching".

Upvotes: 2

Related Questions