Reputation: 4066
I am using Kafka with Spring-boot:
Kafka Producer class:
@Service
public class MyKafkaProducer {
@Autowired
private KafkaTemplate<String, String> kafkaTemplate;
private static Logger LOGGER = LoggerFactory.getLogger(NotificationDispatcherSender.class);
// Send Message
public void sendMessage(String topicName, String message) throws Exception {
LOGGER.debug("========topic Name===== " + topicName + "=========message=======" + message);
ListenableFuture<SendResult<String, String>> result = kafkaTemplate.send(topicName, message);
result.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
@Override
public void onSuccess(SendResult<String, String> result) {
LOGGER.debug("sent message='{}' with offset={}", message, result.getRecordMetadata().offset());
}
@Override
public void onFailure(Throwable ex) {
LOGGER.error(Constants.PRODUCER_MESSAGE_EXCEPTION.getValue() + " : " + ex.getMessage());
}
});
}
}
Kafka-configuration:
spring.kafka.producer.retries=0
spring.kafka.producer.batch-size=100000
spring.kafka.producer.request.timeout.ms=30000
spring.kafka.producer.linger.ms=10
spring.kafka.producer.acks=0
spring.kafka.producer.buffer-memory=33554432
spring.kafka.producer.max.block.ms=5000
spring.kafka.bootstrap-servers=192.168.1.161:9092,192.168.1.162:9093
Let's say I have sent 10 times 1000 messages in topic my-test-topic
.
8 out of 10 times I am successfully getting all messages in my consumer but sometimes I am getting this below error:
2017-10-05 07:24:11, [ERROR] [my-service - LoggingProducerListener - onError:76] Exception thrown when sending a message with key='null' and payload='{"deviceType":"X","deviceKeys":[{"apiKey":"X-X-o"}],"devices...' to topic my-test-topic
and org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for my-test-topic-4 due to 30024 ms has passed since batch creation plus linger time
Upvotes: 36
Views: 92208
Reputation: 41
One possible reason is that the Kafka server reached the maximum heap size limit (-Xmx). Under high load, when the garbage collector runs, it can cause the server to take longer to process requests. In my case, after I increased the maximum memory limit for Kafka, the error was resolved.
Upvotes: 0
Reputation: 171
In my case, I had replication factor < min.insync.replicas
. A write could never get enough acknowledgments and therefore timed out.
Recreating the topic with replication factor > min.insync.replicas
fixed it.
Upvotes: 2
Reputation: 7
I resolved this issue by addressing correctly the host spring.kafka.bootstrap-servers with its DNS. Even if the network resolves the IP addresses, it seems to need the DNS.
Upvotes: -1
Reputation: 1576
First clue in the error is 30024 ms has passed
- the configuration spring.kafka.producer.request.timeout.ms=30000
is related. This 30 seconds wait is for filling up the buffer on the Producer side.
When a message is published, it gets buffered at the Producer's side and it will wait for 30s (see config above) to fill up. spring.kafka.producer.batch-size=100000
means 100KB so if the message ingestion load is low, and the buffer doesn't fill up with more messages to 100KB in 30s, you would expect this message.
spring.kafka.producer.linger.ms=10
is used where the ingestion load is high and the producer wants to limit send()
calls to Kafka brokers. This is the duration Producer will wait before sending messages to the broker after batch is ready (i.e. after buffer is filled up to batch size of 100KB).
Solution:
linger.ms
to hold messages longer after the batch is ready. If more time is needed to fill the batch, increase request.timeout.ms
.batch-size
, or increase request.timeout.ms
, or both.Upvotes: 1
Reputation: 3504
There are 3 possibilities:
request.timeout.ms
- this is the time that Kafka will wait for whole batch to be ready in buffer. So in your case if there are less than 100 000 messages in buffer, timeout will occur. More info here: https://stackoverflow.com/a/34794261/2707179batch-size
- related to previous point, it will send batches more often but they will include fewer messages.Upvotes: 20