KafkaPublisher blocks when cluster is down

Question

I am using kafka-clients-0.8.2.1.jar and found that when clusters are not up at all, the send would block. I found more relevant information online Kafka 0.8.2 new producer blocking on metadata and found it's a known issue many people are facing. However i tried some options mentioned there but didn't help: 1. call KafkaProducer.partitionsFor(), it still blocked. 2. setting producer.type to async but it's not recognized.

What i want to achieve is to detect cluster is down, queue up messages(with a max limit) and send when the cluster is back. If that's too complex, at least it won't block because it cause the application to queue up all messages and run out of memory.

Code:

System.out.println("props:"
producer = new KafkaProducer<String, String>(props);


producer.partitionsFor(record.topic());
producer.send(record, new Callback() {

Log:

props:{queue.size=1000, reconnect.backoff.ms=10000, request.timeout.ms=1000, bootstrap.servers=tstaapp001:59092,ewdlxsrv283:59092,devcapp001:59092, value.serializer=org.apache.kafka.common.serialization.StringSerializer, request.required.acks=1, buffer.memory=33554432, retries=0, producer.type=async, key.serializer=org.apache.kafka.common.serialization.StringSerializer, linger.ms=1, topic.metadata.refresh.interval.ms=1000, batch.size=16384, timeout.ms=10000}
20:44:33.584 [RTEMPool-1-RTEMThread-4] DEBUG o.a.k.c.producer.internals.Metadata - Updated cluster metadata version 1 to Cluster(nodes = [Node(tstaapp001, 59092), Node(devcapp001, 59092), Node(ewdlxsrv283, 59092)], partitions = [])
20:44:33.603 [kafka-producer-network-thread | producer-1] DEBUG o.a.k.c.producer.internals.Sender - Starting Kafka producer I/O thread.
20:44:33.603 [RTEMPool-1-RTEMThread-4] WARN  o.a.k.c.producer.ProducerConfig - The configuration topic.metadata.refresh.interval.ms = null was supplied but isn't a known config.
20:44:33.603 [RTEMPool-1-RTEMThread-4] WARN  o.a.k.c.producer.ProducerConfig - The configuration request.timeout.ms = null was supplied but isn't a known config.
20:44:33.603 [RTEMPool-1-RTEMThread-4] WARN  o.a.k.c.producer.ProducerConfig - The configuration producer.type = null was supplied but isn't a known config.
20:44:33.604 [RTEMPool-1-RTEMThread-4] WARN  o.a.k.c.producer.ProducerConfig - The configuration request.required.acks = null was supplied but isn't a known config.
20:44:33.604 [RTEMPool-1-RTEMThread-4] WARN  o.a.k.c.producer.ProducerConfig - The configuration queue.size = null was supplied but isn't a known config.
20:44:33.604 [RTEMPool-1-RTEMThread-4] DEBUG o.a.k.clients.producer.KafkaProducer - Kafka producer started
20:44:42.958 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:42.959 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Init connection to node -2 for sending metadata request in the next iteration
20:44:42.959 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Initiating connection to node -2 at ewdlxsrv283:59092.
20:44:42.964 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.063 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.163 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.264 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.364 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.464 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.564 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.664 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.764 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.864 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:43.964 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:44.064 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:44.164 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:44.264 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:44.364 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:44.464 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -2
20:44:44.479 [kafka-producer-network-thread | producer-1] WARN  o.a.kafka.common.network.Selector - Error in I/O with null
java.net.ConnectException: Connection refused: no further information

Answer 1

The new producer API will block when its internal buffer is full (this is not well documented). You can add the property block.on.buffer.fullto false, then you will get a BufferExhaustedException.

See http://kafka.apache.org/082/documentation.html#producerapi