CODEWITHSUNDEEP
javaapache-sparkapache-spark-datasetjava-pair-rdd
Manikandan Balasubramanian
Manikandan Balasubramanian

Reputation: 1089

How to convert Dataset into JavaPairRDD?

There are methods to convert Dataset to JavaRDD .

Dataset<Row> dataFrame;
JavaRDD<String> data = dataFrame.toJavaRDD();

Is there any other ways to convert Dataset into javaPairRDD<Long, Vector>?

Upvotes: 5

Views: 7762

Answers (1)

abaghel
abaghel

Reputation: 15297

You can use PairFunction like below. Please check the index of element in your Dataset. In below sample index 0 has long value and index 3 has Vector.

JavaPairRDD<Long, Vector> jpRDD = dataFrame.toJavaRDD().mapToPair(new PairFunction<Row, Long, Vector>() {
    public Tuple2<Long, Vector> call(Row row) throws Exception {
        return new Tuple2<Long, Vector>((Long) row.get(0), (Vector) row.get(3));
    }
});

Upvotes: 10

Related Questions