CODEWITHSUNDEEP
javaperformanceapache-kafkaapache-kafka-streams
Prashanth
Prashanth

Reputation: 21

Performance Issue: Latency Spike happens sometimes in Kafka Streams

I am doing Performance Testing in Kafka Streaming. I created a simple Streams API with Transformer.

  // Stream data from input topic
  builder.stream(Serdes.String(), Serdes.String(), inTopic)
         // convert csv data to avro
         .transformValues(new TransformSupplier())
         // post converted data to output topic
         .to(Serdes.String(), Serdes.ByteArray(), outTopic);

I am using inTopic with 10 partitions and outTopic with 1 partition. I am seeing latency is good and around ~4-6 ms . But, I am facing sudden spike in the latency sometimes and it reaches even upto ~60 - 1000 ms. Then after few seconds, it gradually dropped latency down back to ~4-6 ms. This results in the average latency for my whole experiment to ~67 ms.

What could be the reason for the sudden spike? Suggest me some performance tuning parameters if any.

Note: I have provided default StreamsConfig only.

Upvotes: 0

Views: 939

Answers (1)

Hashcon
Hashcon

Reputation: 595

After some amount of msgs producing, the data should be flushed to the disk.

This may cause the phenomenon you observed.

Please refer to the "log.flush.interval.messages" of kafka configuration: Link

After in prod, I do not recommend you to change this property to improve. You should change your system conf:

/proc/sys/vm/dirty_background_ratio
/proc/sys/vm/dirty_ratio

To improve the efficiency of your own msgs flush

Upvotes: 1

Related Questions