CODEWITHSUNDEEP
javascalaapache-spark
Sandeep540
Sandeep540

Reputation: 977

Scala Error for Hashtable[String, String]

I am writing a small UDF 

  val transform = udf((x: Array[Byte]) => {

    val mapper = new ObjectMapper() with ScalaObjectMapper
    val stream: InputStream  = new ByteArrayInputStream(x);
    val obs = new ObjectInputStream(stream)

    val stock = mapper.readValue(obs, classOf[util.Hashtable[String, String]])
    stock
  })

Where in I get error

java.lang.UnsupportedOperationException: Schema for type java.util.Hashtable[String,String] is not supported
  at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:809)
  at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:740)
  at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
  at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:926)
  at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:49)
  at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:739)
  at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:736)
  at org.apache.spark.sql.functions$.udf(functions.scala:3898)
  ... 59 elided

Can anyone help in understanding why this is coming?

Upvotes: 0

Views: 144

Answers (1)

Oli
Oli

Reputation: 10406

The error you get just means that Spark does not understand java hash tables. We can reproduce your error with this simple UDF.

val gen = udf(() => new java.util.Hashtable[String, String]())

Spark tries to create a DataType (to put in a spark schema) from a java.util.Hashtable, which it does not know how to do. Spark understands scala maps though. Indeed the following code

val gen2 = udf(() => Map("a" -> "b"))
spark.range(1).select(gen2()).show()

yields

+--------+
|   UDF()|
+--------+
|[a -> b]|
+--------+

To fix the first UDF, and yours by the way, you can convert the Hashtable to a scala map. Converting a HashMap can be done easily with JavaConverters. I do not know of any easy way to do it with a Hashtable but you can do it this way:

import collection.JavaConverters._
val gen3 = udf(() => {
    val table = new java.util.Hashtable[String, String]()
    table.put("a", "b")
    Map(table.entrySet.asScala.toSeq.map(x => x.getKey -> x.getValue) :_*)
})

Upvotes: 3

Related Questions