CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
GodBlessYou
GodBlessYou

Reputation: 649

kafka connect exception, Replication factor: 3 larger than available brokers: 1

kafka connect 5.4 only one connector, one worker and use connect-distributed.

Below is the error message:

[2020-06-22 19:09:58,700] ERROR [Worker clientId=connect-1, groupId=test-cluster] 
Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:290)
org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'test-connect-offsets'
    at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:262)
    at org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:99)
    at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:128)
    at org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109)
    at org.apache.kafka.connect.runtime.Worker.start(Worker.java:186)
    at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:121)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:277)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
    at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:229)
    ... 11 more
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

Upvotes: 6

Views: 11594

Answers (3)

Taku
Taku

Reputation: 5928

In addition to what Robin Moffatt answered, you also need to set in Docker:

CONNECT_CONFLUENT_TOPIC_REPLICATION_FACTOR: 1

And you may also want to change the Connector's setting like below:

"topic.creation.default.replication.factor": "1"

(This is Debezium's setting)

Upvotes: 1

Robin Moffatt
Robin Moffatt

Reputation: 32080

For Kafka Connect to run in distributed mode it uses three topics that are stored on the Kafka cluster and hold information about configuration etc. You need to set in the Kafka Connect worker properties:

config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

If you're using the Docker image then you need to set the environment variables to override these, which in Docker Compose looks like:

CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"

Ref: Configuring Kafka Connect distributed workers

Upvotes: 24

GodBlessYou
GodBlessYou

Reputation: 649

it works after adding below to config.properties

offsets.topic.replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

Upvotes: -1

Related Questions