CODEWITHSUNDEEP
apache-kafkaksqldb
Artsiom Krukouski
Artsiom Krukouski

Reputation: 57

How to transform a nested array into a table in ksqldb/kafka

I started learning ksqldb for Kafka and ran into a problem, I have a Product stream and the structure is the following:

 ID          | VARCHAR(STRING)
 NAME        | VARCHAR(STRING)
 ...
 INGREDIENTS | ARRAY<STRUCT<NAME VARCHAR(STRING), ISALLERGEN BOOLEAN, RELATIVEAMOUNT DOUBLE>>

And as a result, I want to create a table and get the last state of the product using 'LATEST_BY_OFFSET' function, but the problem is that I can't apply this function to Array or Struct.

Should I EXPLODE each prop and create a separate entry in the table for each ingredient?(I think it's very strange). How can I deal with this situation? Can you tell me if you have any ideas?

Upvotes: 0

Views: 708

Answers (2)

Artsiom Krukouski
Artsiom Krukouski

Reputation: 57

The problem was solved in release 0.25.1 https://github.com/confluentinc/ksql/pull/8878 https://github.com/confluentinc/ksql/commit/adac45855dce1c413073e5cbeb474cb022013a2b

Upvotes: 1

GeoJim
GeoJim

Reputation: 1355

This is a known issue. It is documented here: https://github.com/confluentinc/ksql/issues/5437 and I happen to be working on it presently. Hopefully a solution will be available in an upcoming version!

I'm not sure how your workaround would work given that after you explode the array that there could be several instance of each component of the struct per key...

Upvotes: 1

Related Questions