minio folder under a bucket is not automatically created by kafka s3 sink connector distributed mode

Question

I've set up kafka s3 sink connector in both standalone mode and distributed mode. They both worked fine, however I observe a difference in the behavior on minio storage side.

I specified topics.dir in sink properties to store the parquet files converted from kafka topics. When I run kafka connector in standalone mode, I don't need to manually pre-create topics.dir. The connector will create the topics.dir, then the sub folder with topic name, then store the parquet files in there.

When I switch to kafka connector in distributed mode, it seems everything is completed on connector side without any error, however, on minio, I don't see the topics.dir folder and topic name sub folder get created. So I tried to manually create the topics.dir. As soon as I manually create the topics.dir, all the sub folders (of topic names) and associated parquet files will show up.

This is a bit inconvenient because you won't know if the connector works normally or not (it actually did) until you manually create the topics.dir folder. Also this manual step breaks the pipeline.

Does anyone know what could be the trigger of this behavior change and how to fix it?

Thanks.

Answer 1

it turns out that there's a delay for these folders to show up on minio GUI.