CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Mehran Prs
Mehran Prs

Reputation: 529

Enable kafka source connector idempotency

How can I enable Kafka source connector idempotency feature?

I know in confluent we can override producer configs by producer.* properties in the worker configuration, but how about Kafka itself? is it the same?

After setting these configs where can I see applied configs for my connect worker?

Upvotes: 1

Views: 859

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191733

Confluent doesn't modify the base Kafka Connect properties.

For configuration of the producers used by Kafka source tasks and the consumers used by Kafka sink tasks, the same parameters can be used but need to be prefixed with producer. and consumer. respectively

Starting with 2.3.0, client configuration overrides can be configured individually per connector by using the prefixes producer.override. and consumer.override. for Kafka sources or Kafka sinks respectively

https://kafka.apache.org/documentation/#connect_running

However, Kafka Connect sources aren't idenpotent - KAFKA-7077 & KIP-308

After setting these configs where can I see applied configs for my connect worker

In the logs, it should show the ProducerConfig or ConsumerConfig when the tasks start

Upvotes: 1

Related Questions