CODEWITHSUNDEEP
amazon-s3apache-kafkaapache-kafka-connect
unknown
unknown

Reputation: 53

Confluent S3 Connector Clarification

We are using confluent's s3 connector to send avro data from a topic to s3. We have 3 broker nodes and on all 3 we have confluent s3-connector running. In the configuration file of connector we have two topics and tasks.max=1. I am new to kafka and I have following doubts:

  1. Since we have overall three s3-connectors, how they are reading from each topic (each topic has 3 partitions and 2 replication factor). Are they considered as three different consumers reading from same topic or all these consumers come under a single consumer group and read data in parallel?
  2. We have two topics in each connector. Do they launch different threads to read data from both the topics in parallel or do they consume sequentially (read from a topic at a time)?

Upvotes: 0

Views: 58

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191874

tasks.max=1

First, set that to the number of total partitions.

Replication factor doesn't matter. Consumers can only ever read from one partition at a time.

Connect forms a consumer group. That is the basic design for any Kafka consumer client. They read in parallel, depending on all your other properties.

Sounds like you are running connect-standalone, and not connect-distributed, however

If you have 3 machines, obviously use distributed mode

And yes, tasks and threads are funtionally equivalent, with the difference being that tasks will rebalance , while threads are logically only on a single machine.

Upvotes: 0

Related Questions