CODEWITHSUNDEEP
apache-sparkvectorapache-spark-mllibapache-spark-ml
Nagireddy Hanisha
Nagireddy Hanisha

Reputation: 1440

VectorUDT usage

I have to get the datatype and do a case match and convert it to some required format. But the usage of org.apache.spark.ml.linalg.VectorUDT is showing VectorUDT is private. Also I specifically need to use org.apache.spark.ml.linalg.VectorUDT and not org.apache.spark.mllib.linalg.VectorUDT. Can someone suggest how to go about this?

Upvotes: 2

Views: 1520

Answers (1)

zero323
zero323

Reputation: 330163

For org.apache.spark.ml.linalg types you should specify schema using org.apache.spark.ml.linalg.SQLDataTypes which provide singleton instances of the private UDT types:

  • MatrixType for matrices (org.apache.spark.ml.linalg.Matrix).

    scala> org.apache.spark.ml.linalg.SQLDataTypes.MatrixType.getClass
res0: Class[_ <: org.apache.spark.sql.types.DataType] = class org.apache.spark.ml.linalg.MatrixUDT

  • VectorType for vectors (org.apache.spark.ml.linalg.Vector).

    scala> org.apache.spark.ml.linalg.SQLDataTypes.VectorType.getClass
res1: Class[_ <: org.apache.spark.sql.types.DataType] = class org.apache.spark.ml.linalg.VectorUDT

Upvotes: 4

Related Questions