CODEWITHSUNDEEP
amazon-s3apache-kafkaapache-kafka-connectaws-msks3-kafka-connector
Shashwat Awasthi
Shashwat Awasthi

Reputation: 27

Configuring folder for path kafka-s3-connector from message

I have data of below mentioned format in AWS MSK. I want to push this data into AWS s3 bucket but I need to make the folder structure from the values of variables in the message. How do I do this using kafka-s3-connector?

I want this data to be stored with path

bucket-name/{schema-name}/{table-name}/{year}/{month}/{date}

{
    "data": {
        "id": 630
    },
    "beforeData": {
        "id": 630
    },
    "metadata": {
        "timestamp": "2024-10-24T06:09:56.659338Z",
        "record-type": "data",
        "operation": "update",
        "partition-key-type": "schema-table",
        "schema-name": "customers_dev",
        "table-name": "customer_con",
        "transaction-id": 3823339587959
    }
}

Upvotes: 0

Views: 97

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

You'll need to write your own Partitioner implementation in Connect.

It's not possible to extract/route records to storage based on both field and Date. See https://github.com/confluentinc/kafka-connect-storage-common/pull/251

Upvotes: 1

Related Questions