CODEWITHSUNDEEP
apache-kafkalatencykafka-producer-apimirroring
grinferno
grinferno

Reputation: 534

Huge amounts of lag when using kafka mirrormaker

I have set up a MirrorMaker cluster that consumes topics from clusters around the world.

The issue is the latency between the mirror cluster (located in the EU) and a source cluster (located in the US) creates a massive spike in offset lag.

The mirrors are consuming 9 topics, each one consisting of 24 partitions.

Mirror Settings

/opt/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config /opt/kafka/config/us1.consumer.properties --num.streams 48 --producer.config /opt/kafka/config/glb.producer.properties --whitelist="a,b,c,d,e,f,g,h,i"

The producer.properties Settings

bootstrap.servers=localhost:9092
acks=-1
retries=2147483647
client.id=us.mm.producer
batch.size=200
linger.ms=10

I am playing around with increasing the batch size and introducing linger to try and increase throughput, but so far my efforts have failed.

What is the best way to work the ideal batch size, and is introducing a linger.ms going to be useful with such high latency rates?

Apologies if this seems vague, but I am new to Kafka, and tuning a server is proving difficult.

Upvotes: 0

Views: 5586

Answers (3)

Jack
Jack

Reputation: 165

batch.size=200 seems low, please be aware that 200 is maximum number of bytes that will be included in a batch.

Upvotes: 0

srikanth daggumalli
srikanth daggumalli

Reputation: 1

First - How much latency you see ?

Latency also depends upon the data throughput rate. In my case for 10k Events per second, with 4 KB record(Size of the record matters too). I have observed producer avg latency as 1.5 ms and max latency as 3.5 ms and with 100k EPS, I have observed producer avg latency as 3.5 ms and max latency close to 30 Seconds (which is very high)

It depends upon many factors. Overall Latency = Network Latency (can be checking using Ping one of the US box and it will give you RTT) + Mirror Maker Latency (Confluent replicator cliams that it introduces 16ms and we can expect more or less with Apache Kafka mirror maker)

Also check, 

Producer buffer memory - default is 32 MB.
acks recommendation is 1 (i.e. leader+ 1 follower) o.w it will more delays.
socket buffer
send.buffer.bytes
receive.buffer.bytes
num.network.threads
num.io.threads

I didn't get retries in your config. Any compression enabled ?

Upvotes: 0

Michal Borowiecki
Michal Borowiecki

Reputation: 4314

For tuning producer settings I recommend starting by following the methodology described in this slide deck: https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600

However, since mirror maker is running in the target cluster, I suspect that the bottleneck might be consumption from the source cluster (US->EU), not producing the target cluster, which is co-located with mirror maker, if I understood correctly.

Upvotes: 1

Related Questions