CODEWITHSUNDEEP
scalaapache-sparkkryo
Angel F O
Angel F O

Reputation: 76

Spark Kryo Registrator

I'm using kryo as serializer in spark 2.1 I have set registration as required but I have facing a problem, I can't register this class:

org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[]

I have tried with classOf[Array[Class.forName("org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation").type]] and other combinations but I don't succeed in the registration of this class.

I have tried as well creating a java class like register and registering the class as org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[].class;

Any clue is appreciated, Thanks.

Upvotes: 0

Views: 558

Answers (1)

Angel F O
Angel F O

Reputation: 76

I have solved my question, I've used:

kryo.register(ClassTag(Class.forName("org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableBlockLocation")).wrap.runtimeClass)

It allows register for the class name at runtime wrapped in an array.

Upvotes: 2

Related Questions