CODEWITHSUNDEEP
apache-sparkapache-spark-sqlapache-spark-datasetapache-spark-encoders
Manikandan Balasubramanian
Manikandan Balasubramanian

Reputation: 1089

How to set Encoder for Row, LabeledPointData in Spark?

How to set Encoders for LabeledPointData which is combination of Double, Vectors of Double. How to set Encoders for creating DataFrame? 

public static Dataset<LabeledPoint> convertRDDStringToLabeledPoint(Dataset<String> data,String delimiter) {
    Dataset<LabeledPoint> labeledPointData = data.map(
            (data1)->{
                String splitter[] = data1.split(delimiter);
                double[] arr = new double[splitter.length - 1];
                IntStream.range(0,arr.length).forEach(i->arr[i]=Double.parseDouble(splitter[i+1]));
                return new LabeledPoint(Double.parseDouble(splitter[0]), Vectors.dense(arr));
            },Encoders.???);
    return labeledPointData;
}

Upvotes: 0

Views: 456

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74739

LabeledPoint is a case class in Scala so I think it's Encoders.product[LabeledPoint].

(I don't know how to write it in Java)

Upvotes: 1

Related Questions