Reputation: 11
I have a set of tables in Oracle, each with the same columns. I want to move this data from Oracle to Snowflake using Kafka streams. One of the columns is a BLOB. I do not need this column in Snowflake.
When configuring the source Debezium connector, I saw there is this transformation called "ExtractNewRecordState" which has the methods "add.fields", which works nicely to add columns to the message, which will end up in Snowflake next to the table columns from Oracle. I was looking for a "remove.fields" or "ignore.fields" but there isn't one.
What I did so far:
I found this instead "drop.fields.header.name" but it is about the header. I can't just pass Oracle columns to it.
The AI tools suggested to use some Kafka transformation tools but they didn't work: "transforms.filter.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", with "whitelist" option to enumerate the columns I want. Or "transforms.extract.type": ""org.apache.kafka.connect.transforms.ExtractField$Value", with the "extract" option and ,again, enumerate the columns I want.
Does anyone has any suggestions? I need a hint. Can this be done at the "Sink" level? (meaning, should it be done when consuming messages from Kafka and streaming into Snowflake?)
Upvotes: 1
Views: 219