CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqlapache-spark-ml
Alexey Svyatkovskiy
Alexey Svyatkovskiy

Reputation: 646

Issue with VectorUDT when using Spark ML

I am writing an UDAF to be applied to a Spark data frame column of type Vector (spark.ml.linalg.Vector). I rely on spark.ml.linalg package so that I do not have to go back and forth between dataframe and RDD.

Inside the UDAF, I have to specify a data type for the input, buffer, and output schemas:

def inputSchema = new StructType().add("features", new VectorUDT())
def bufferSchema: StructType =
    StructType(StructField("list_of_similarities", ArrayType(new VectorUDT(), true), true) :: Nil)

override def dataType: DataType = ArrayType(DoubleType,true)

VectorUDT is what I would use with spark.mllib.linalg.Vector: https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala

However, when I try to import it from spark.ml instead: import org.apache.spark.ml.linalg.VectorUDT I get a runtime error (no errors during the build): 

class VectorUDT in package linalg cannot be accessed in package org.apache.spark.ml.linalg

Is it expected/can you suggest a workaround?

I am using Spark 2.0.0

Upvotes: 8

Views: 8144

Answers (1)

nedim
nedim

Reputation: 1857

In Spark 2.0.0, the proper way to go is to use org.apache.spark.ml.linalg.SQLDataTypes.VectorType instead of VectorUDT. It was introduced in this issue.

Upvotes: 28

Related Questions