Reputation: 23825
I have a Kafka message that looks like the below.
{
“prop1”: “val1”,
“prop2”: “val2”,
“sub_msg”: {
“prop3”: “val3”,
“prop4”: “val4”,
},
“prop5”: “val5”
}
And I need to flatten it. Like the below.
{
“prop1”: “val1”,
“prop2”: “val2”,
“prop3”: “val3”,
“prop4”: “val4”,
“prop5”: “val5”
}
I am looking at the ReplaceField SMT that is able to to replace field names in a flat structure.
My question is - can this SMT handle nested structures as well? Or - is there a "flattener" SMT available out of the box?
EDIT (1)
I can see the flatten SMT but it adds the "path" of the nested object - If I could disable the "path" concatenation - it could work for me. I cant see anyway to disable it in the docs.
EDIT (2)
My current direction is to extend the flattener and make it possible not to concatenate field names.
New configuration fields:
With the two config attributes above one can set the SMT behaviour regarding field names concatenation and control what should happen in case of duplicates
Upvotes: 0
Views: 685
Reputation: 192023
ReplaceField can only be used on top-level fields.
You'd need to combine the flatten transform with the rename transform to be able to remove the sub_msg
prefixes.
Upvotes: 1