CODEWITHSUNDEEP
apache-sparkapache-spark-sqlkryo
Georg Heiler
Georg Heiler

Reputation: 17724

spark kryo registration class missing

When forcing kryo registration via spark.kryo.registrationRequired", "true" in spark 2.2 my error is

Class is not registered: org.apache.spark.sql.catalyst.InternalRow[]

even when using:

.registerKryoClasses(
          Array(classOf[scala.collection.mutable.WrappedArray.ofRef[_]],
                classOf[org.apache.spark.sql.catalyst.InternalRow])
        )

Obviously InternalRow[] would not compile in scala. What am I missing here?

Upvotes: 1

Views: 1133

Answers (1)

Georg Heiler
Georg Heiler

Reputation: 17724

How to register InternalRow with Kryo in Spark is leading me to the correct answer.

classOf[Array[org.apache.spark.sql.catalyst.InternalRow]]

is required. However then, further spark classes need to be added as you move forward.

Upvotes: 1

Related Questions