CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
balderman
balderman

Reputation: 23825

Kafka Connect SMT - flattening a message

I have a Kafka message that looks like the below.

{
   “prop1”: “val1”,
   “prop2”: “val2”,
   “sub_msg”: {
       “prop3”: “val3”,
       “prop4”: “val4”,
    },
   “prop5”: “val5”
}

And I need to flatten it. Like the below.

{
   “prop1”: “val1”,
   “prop2”: “val2”,
   “prop3”: “val3”,
   “prop4”: “val4”,
   “prop5”: “val5”
}

I am looking at the ReplaceField SMT that is able to to replace field names in a flat structure.
My question is - can this SMT handle nested structures as well? Or - is there a "flattener" SMT available out of the box?

EDIT (1)

I can see the flatten SMT but it adds the "path" of the nested object - If I could disable the "path" concatenation - it could work for me. I cant see anyway to disable it in the docs.

EDIT (2)
My current direction is to extend the flattener and make it possible not to concatenate field names.

New configuration fields:

  1. concatenate.field.names | boolean | default: true
  2. duplicate.field.strategy | Enum (FirstSeenWins,LastSeenWins) | default: LastSeenWins

With the two config attributes above one can set the SMT behaviour regarding field names concatenation and control what should happen in case of duplicates

Upvotes: 0

Views: 685

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 192023

ReplaceField can only be used on top-level fields.

You'd need to combine the flatten transform with the rename transform to be able to remove the sub_msg prefixes.

Upvotes: 1

Related Questions