CODEWITHSUNDEEP
apache-kafka-connect
M. Ferris
M. Ferris

Reputation: 361

How to transform more than 1 field in Kafka Connect?

I am using a Kafka Connect Sink config to get data from a topic and persist to an Oracle DB. Works like a champ, and I'm doing a transformation on a timestamp column that comes in via an Avro schema as a long, and I then transform to an Oracle Timestamp column. 

"transforms": "TimestampConverter",
"transforms.TimestampConverter.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.TimestampConverter.format": "mm/dd/yyyy HH:mm:ss",
"transforms.TimestampConverter.target.type": "Timestamp",
"transforms.TimestampConverter.field": "created_ts"

But, I can't figure out how to do this on multiple timestamps. That is, in addition to the created_ts, I also have an updated_ts I need to transform.

I tried this: 

"transforms.TimestampConverter.field": "created_ts, updated_ts"

Does not work, nor can I repeat the whole block for the other field, because Connect only allows 1 same-named entry.

Lastly, I tried this: 

"transforms.TimestampConverter.field.1": "created_ts",
"transforms.TimestampConverter.field.2": "updated_ts"

Upvotes: 4

Views: 3413

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191723

You would add 2 transforms

"transforms": "CreatedConverter,UpdatedConverter",
"transforms.CreatedConverter.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value"
"transforms.CreatedConverter.field": "created_ts",
...
"transforms.UpdatedConverter.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value"
"transforms.UpdatedConverter.field": "updated_ts"
...

Upvotes: 6

Related Questions