CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
TheDataGuy
TheDataGuy

Reputation: 3118

Delete the kafka connect topics without stopping the process

I was running a Kafka connect worker in distributed mode. (it's a test cluster), I wanted to reset the default connect-* topics,so without stopping the worker I removed, then After the worker restart, I'm getting this error.

ERROR [Worker clientId=connect-1, groupId=debezium-cluster1] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:324)

org.apache.kafka.common.config.ConfigException: 
Topic 'connect-offsets' supplied via the 'offset.storage.topic' property is required to have 'cleanup.policy=compact' to guarantee consistency and durability of source connector offsets, 
but found the topic currently has 'cleanup.policy=delete'. 
Continuing would likely result in eventually losing source connector offsets and problems restarting this Connect cluster in the future. 
Change the 'offset.storage.topic' property in the Connect worker configurations to use a topic with 'cleanup.policy=compact'.

Upvotes: 0

Views: 2406

Answers (2)

Barak
Barak

Reputation: 3066

Deleting the internal topics while the workers are still running sounds risky. The workers have internal state, which now no longer matches the state in the Kafka brokers.

A safer approach would be to shut down the workers (or at-least shut down all the connectors), delete the topics, and restart the workers/connectors.

Upvotes: 1

Robin Moffatt
Robin Moffatt

Reputation: 32110

It looks like the topics got auto-created, perhaps by the workers when you deleted them mid-flight.

You could manually apply the configuration change to the topic as suggested, or you could also specify a new set of topics for the worker to use (connect01- for example) and let the workers recreate them correctly.

Upvotes: 0

Related Questions