CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Mark
Mark

Reputation: 611

Kafka Connect connector limiting to one sink task per topic

I am running a Kafka Connect cluster (locally with 1 worker Docker Compose) with my own custom Sink plugin. I have several topics that I want to consume within the connector: topicA, topicB, topicC, all having a single partition each.

A subset of my configuration for my connector when it is started is as follows:

...
"topics": "topicA,topicB,topicC",
"tasks.max": 3,
...

With this configuration I would expect Kafka Connect to assign one topic per sink task but alas this is not what I'm seeing. What happens in practice is that SinkTask::open is called for each task with all of the topics assigned. Each sink task is also seeing records come in from multiple topics, not just one. I can confirm that all tasks are indicated as running.

Is there anything that I need to enable, other than the above configuration, to enable Kafka Connect to map exactly one sink task to one topic? Is it a result of only having one worker and Kafka Connect not breaking apart the tasks into separate threads on the same worker?

Upvotes: 1

Views: 955

Answers (2)

Mark
Mark

Reputation: 611

I ended up using the SinkTaskContext's pause() and resume() methods creatively. Kafka Connect will conveniently call open(...) on the topic-partitions that the consumer is subscribed to and within that hook is a good opportunity to pause topics that you're not interested in.

See https://kafka.apache.org/26/javadoc/org/apache/kafka/connect/sink/SinkTaskContext.html

Upvotes: 0

OneCricketeer
OneCricketeer

Reputation: 191710

In my experience, the only way to do what you're asking for is to create 3 different connectors with only one topic.

More specifically, giving a list or topic pattern will simply subscribe any consumer that's available to that whole list.

I've personally not seen any way to accurately control how tasks get rebalanced or assigned to topics or their partitions. You can modify the consumer client settings in Connect, but that's only after the tasks are assigned.

Upvotes: 1

Related Questions