CODEWITHSUNDEEP
apache-kafkatransformksqldb
shakeel
shakeel

Reputation: 901

Transform data in ksql

I am trying to convert data from one format to another format (one schema to another schema).

Example : 

payload = {
    'a' : 'a1',
    'b' : 'b1'
}

and i want to transform this payload into another form let us say

payload_transform = {
   'a':{
      'b' : 'b1'
    }
    'c' : 'a1'
}

consider that data(payload) is coming from Kafka, i want to see payload_transform in consumer that with transformation

is it possible with ksql ?

Updated :

Can we do one level : 

payload = {
    'a' : 'a1',
    'b' : 'b1'
}

to

payload = {
    'confluent' : 'a1',
    'b' : 'b1'
}

and can we add conditions ?

For example : if 'b' key present in payload generate 

payload = {
        'confluent' : 'a1',
        'b' : 'b1'
    }

otherwise : 

payload = {
        'kafka' : 'a1',
        'b' : 'b1'
    }

Upvotes: 1

Views: 559

Answers (1)

Robin Moffatt
Robin Moffatt

Reputation: 32100

Whilst KSQL does support un-nesting JSON (with EXTRACTJSONFIELD), it does not currently (March 2018 / version 0.5) support building nested structures. It also does not currently support nested Avro.

Updated response to updated question:

  • You can rename fields, simply using the SQL AS clause:

    SELECT A AS NEW_COL, B FROM INPUT_STREAM

Can you describe more about what you're trying to do here? Renaming a field conditionally doesn't make sense in the example you've given. Perhaps also give KSQL a try and see what works for you.

Upvotes: 3

Related Questions