CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectmongodb-kafka-connector
oy121
oy121

Reputation: 103

Kafka connect sink connector with multiple one partitioned topics

We would like to use the Kafka connect sink connector in order to copy messages from Kafka to Mongo DB. In our use case, we have multiple topics, with one partition each (the name of the topics can be regexed, for example, topic.XXX.name). The number of these topics is increasing continuously. I wonder if the Kafka connect architecture fits this use case. If so, how can it be configured the gain high scaleability and parallelism? What will be the tasks.max? Number of workers?

Upvotes: 1

Views: 1631

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

Kafka Connect is flexible; the answer is as many as you need.

The number of running tasks per connect worker is mostly only limited by the JVM heap size of each worker. Adding more workers will allow you to have more total active connectors.

For sink connectors, however, you can only have as many total tasks as total topic partitions being consumed.

One thing to worry about, though, is frequent consumer group rebalancing as you add more and more topics. For this reason, it would be recommended to create independent connectors for any critical data

Upvotes: 0

Related Questions