CODEWITHSUNDEEP
apache-flink
jason zhang
jason zhang

Reputation: 451

Apache Flink:How does the groupBy/ sortBy transform support on the dynamic tuples in dataset api?

I use java.util.Map as data type which is not supported by position or expression keys, so how can I to group a java.util.Map based dataset if the number of group fields is more than 25?

Example code show as below:

Map<String,Object> input1 = new HashMap<>();
for (int i=0; i<30; i++){
    input.put("groupField" + i,"value"+i);
}
input1.put("quantity",200);
Map<String,Object> input2 = new HashMap<>();
for (int i=0; i<30; i++){
    input2.put("groupField" + i,"value"+i);
}
input2.put("quantity",200);

DataSet<Map<String,Object>> input = env.fromElements(input1,input2);
//how can i group this map based dataSet and aggregate on the 'quantity' field if the number of grouping fields is more than 25?

Upvotes: 2

Views: 817

Answers (1)

Fabian Hueske
Fabian Hueske

Reputation: 18997

I would recommend to use a custom class as key type for group and sort operations, i.e., return objects of this key class from a KeySelector. A custom class can be used as key if it implements the java.lang.Comparable interface and the Object.hashCode() method correctly.

For example, your key type could be a simple wrapper around an java.util.ArrayList to support arbitrary many fields.

Upvotes: 1

Related Questions