CODEWITHSUNDEEP
apache-kafkakafka-consumer-apiapache-kafka-connect
omer bar lev
omer bar lev

Reputation: 401

Batch creation of Kafka Connector

I'm trying to create 1000 Connectors, each one with one task, his own consumer group and unique topic in my Kafka Kubernetes cluster (My end goal after creating the connectors, is sending a lot of requests to the connectors' topics and measure performances for our implemented connector sink).

Every creation triggers rebalancing across the cluster which "blocks" the Connector RestAPI (returns 409 for everything) and shutting down tasks.

Therefore I have three questions:

  1. Is the rebalancing a sort of downtime for the connector (As I said, there are tasks shutting down and restart while rebalancing and one task for connector)?
  2. Can I configure the rebalancing schedule?
  3. Is there a way of creating Connectors in batches so it would be fast (say creating 100 connectors in less than one second) and won't cause downtime (if the answer for the first question is yes)?

Upvotes: 0

Views: 227

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191710

One way around the problem would be to start 1000 Connect Clusters (say, via Docker Orchestration), all with one or few Connectors.

There isn't a way around the rebalancing. You're adding consumers to the same consumer group, so that'll always rebalance.

Rather than running one task per connector, I would suggest grouping multiple topics/tasks together than share similar configurations, that way you limit how much rebalancing would be done.

Upvotes: 1

Related Questions