CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectconfluent-platform
Nilkanth Patel
Nilkanth Patel

Reputation: 31

Kafka sink connect - How to create one task per topic(table)

We have implemented a kafka sink connector for out product named Ampool ADS which ingestt data from kafka topics to corresponding ampool tables. Topics and Tables are mapped by their names.

I Need to handle individual topic (ingestion from topic ---> table) into a dedicated sink task.

So for example, if my config contains 3 different topics (topic1, topic2, topic3), Sink connector should create 3 different sink tasks, each doing (per table) dedicated ingestion to their respective/mapped table in parallel.

NOTE: Reason behind handling an individual topic into a dedicated sink task is its simple to use RetriableException mechanism if specific table are offline/not-created. Only a individual topic/table records will get replayed after configured time-interval.

Is this possible with kafka connect framework, if so how..?

Upvotes: 0

Views: 1278

Answers (1)

Gwen Shapira
Gwen Shapira

Reputation: 5158

If you set the number of tasks to be equal to the number of partitions (and I think you can do this from the connector code - when creating the configuration), then each task will get exactly one partition.

Upvotes: 1

Related Questions