CODEWITHSUNDEEP
apache-kafka
mukulbudania
mukulbudania

Reputation: 301

Kafka Producer not able to send messages

I am very new to Kafka.

Using Kafka 0.11

Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor')

I get the above error on sending a message for a topic

kafka-topics --zookeeper localhost:2181 --topic test --describe
Topic:test1 PartitionCount:1    ReplicationFactor:1 Configs:
Topic: test1    Partition: 0    Leader: 0   Replicas: 0 Isr: 0

Upvotes: 13

Views: 13254

Answers (3)

Mansur Ul Hasan
Mansur Ul Hasan

Reputation: 3606

This means your cluster have default replication factor setting to some numbers to override this you need to edit server.properties and add replication factor parameter with your choice value 

offsets.topic.replication.factor=1

In my case i wanted to run single node kafka with single node zookeeper for that case you need to create topic with replication factor 1 otherwise you will gent an error 

mansoor@c2dkb05-usea1d:~$ ./bin/kafka-topics.sh --create --zookeeper zookeeper-svc:2181 --replication-factor 2 --partitions 2 --topic mqttDeviceEvents

Error while executing topic command : Replication factor: 2 larger than available brokers: 1.
[2020-06-18 14:39:46,533] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1.

the correct way to create topic is when you have single node kafka 

mansoor@c2dkb05-usea1d:$ ./bin/kafka-topics.sh --create --zookeeper zookeeper-svc:2181 --replication-factor 1 --partitions 2 --topic mqttDeviceEvents
Created topic mqttDeviceEvents.

Upvotes: 1

ppatierno
ppatierno

Reputation: 10065

How are you starting the broker ? What is the server.properties file. The one provided with the downloaded package should have the following line :

offsets.topic.replication.factor=1

Just to be clear the error you see is not related to the topic you are trying to publish. Today, Kafka doesn't save topic offsets for consumers in Zookeeper anymore but in "internal topics" with name __consumer_offsets. Of course, if you have 1 broker you can't have a replication factor of 3. So I'd like to take a look at your server.properties. If the above property is missing, the default is 3.

Upvotes: 22

daemonsl
daemonsl

Reputation: 452

In my case, my error is also similar.

ERROR [KafkaApi-2] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

Cluster : 2 Brokers (ID=1,ID=2) with hostname-1 and hostname-2
Kafka version : 1.0.1

listeners=PLAINTEXT://:9090,SSL://:9091,PLAINTEXT_EXT://:9092,SSL_EXT://:9093,SASL_SSL://:9094,SASL_PLAINTEXT://:9095

and both broker server.properties is set to offsets.topic.replication.factor=1

but I configured my advertised hostname as hostname-1 in both broker over protocols which is being used for inter-broker communication ( and thus broker with ID=2 keep on giving above error.

advertised.listeners=PLAINTEXT://hostname-2:9090,SSL://hostname-2:9091,PLAINTEXT_EXT://<EXTERNAL_IP>:9092,SSL_EXT://<EXTERNAL_IP>:9093,SASL_SSL://hostname-1:9094,SASL_PLAINTEXT://hostname-1:9095

correction on SASL_SSL and SASL_PLAINTEXT fixed this error.

PS : SASL_PLAINTEXT is security.inter.broker.protocol in this cluster. This error seems to be related with port availability as well.

Upvotes: 1

Related Questions