CODEWITHSUNDEEP
amazon-s3apache-kafkaapache-kafka-connectconfluent-platforms3-kafka-connector
benjaminedwardwebb
benjaminedwardwebb

Reputation: 133

Can Confluent's S3 Sink Connector for Kafka Connect write topics to a nested (not a top-level) folder in an S3 bucket using `topics.dir`?

Can Confluent's S3 Sink Connector for Kafka Connect write topics to a nested (not a top-level) folder in an S3 bucket using topics.dir?

For example, if I set topics.dir to the value thisistoplevel/thisisnested, will the connector work?

The documentation for the topics.dir configuration property says:

Top level directory to store the data ingested from Kafka.

But it seems like a strange restriction to make. Perhaps the wording "Top level directory" is meant to mean something more like "the top-most directory under which topics will be written".

Maybe someone else has tested this or uses a nested directory in production?

Upvotes: 0

Views: 1137

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

In S3's context, there is no such thing as "nesting" of folders; S3 does not have "folders". It is rather just a string prefix within a bucket.

Yes, the prefix can be as long as you need it, and then topic names will be appended to that path when written

Upvotes: 1

Related Questions