CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
NickSar68
NickSar68

Reputation: 55

Kafka connect transformations ExtractField$Value while preserving more keys

I have the following message structure:

{
    "payload" {
       "a": 1,
       "b": {"X": 1, "Y":2}          
    },
    "timestamp": 1659692671
}

I want to use SMTs to get the following structure:

{
   "a": 1,
   "b": {"X": 1, "Y":2},
   "timestamp": 1659692671
}

When I use ExtractField$Value for payload, I cannot preserve timestamp. I cannot use Flatten because "b"'s structure should not be flatten.

Any ideas? Thanks

Upvotes: 0

Views: 831

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191723

Unfortunately, moving a field into another Struct isn't possible with the built-in transforms. You'd have to write your own, or use a stream-processing library before the data reaches the connector.

Note that Kafka records themselves have a timestamp, so do you really need the timestamp as a field in the value? One option is to extract the timestamp, then flatten, then add it back. However, this will override the timestamp that the producer had set with that value.

Upvotes: 1

Related Questions