CODEWITHSUNDEEP
apache-kafkadebeziumaws-msk
dz902
dz902

Reputation: 5838

Using Debezium MySQL connector with MSK shows "INVALID_REPLICATION_FACTOR"

I'm using Debezium MySQL with MSK, very simple setup. Connection to MySQL (Aurora) is tested fine. Kafka topics creation, listing are both ok.

However, when I run the connector, after a lot of scrolling info I get

WARN [Producer clientId=xxx] Error while fetching metadata with correlation id 1 : {xxx.xxx=INVALID_REP
LICATION_FACTOR} (org.apache.kafka.clients.NetworkClient:1100)

A lot of them keeps showing up and the connector does not work properly.

Upvotes: 1

Views: 964

Answers (1)

dz902
dz902

Reputation: 5838

After quite a while I found out the reason for this is that default replication factor setting on MSK follows Kafka best practice which is 3, but I only created 2 brokers.

The configuration stayed 3 and when connector tried to auto-create a topic with 3 replicas it fails. The strange thing is even when I manually created a topic with replication factor 2, the connector would throw the very same warning.

It seems that the internal topics are always attempted by Debezium connector.

Creating a new revision and set replication factor as 2 solved the problem.

Upvotes: 4

Related Questions