CODEWITHSUNDEEP
javalambdaapache-spark
user3756702
user3756702

Reputation: 215

apache spark using java lambda to restructure key-value pair

I played around with apache spark and I have a key value pair, whose value is an ArrayList and I would like to move one value out of the ArrayList to Key.Position and the Key back into the ArrayList. Is ther a way to do this with lambda expression?

In Python it looks like this

newMap = sourceMap.map(lambda (key,((value1, value2), value3)) : (value1, (key, value2,value3)))

How to do this in Java with lambdas? sourceMap is of the same type as newMap

JavaPairRDD<String, ArrayList<String> newMap = sourceMap.flatMapToPair((a, b) -> ??? )

Upvotes: 2

Views: 924

Answers (1)

user3756702
user3756702

Reputation: 215

Here is my solution for this

JavaPairRDD<String, ArrayList<String>> newRDD = source.mapToPair( lines -> {

        ArrayList<String> values = new ArrayList<String>();

        // restructure
        values.add(lines._1());
        values.add(lines._2()._1().get(0));
        values.add(lines._2()._1().get(2));
        values.add(lines._2()._2());

        Tuple2<String, ArrayList<String>> tuple = new Tuple2<>(lines._2()._1().get(1), values);

        return tuple;
    });

Upvotes: 1

Related Questions