CODEWITHSUNDEEP
apache-sparkspark-graphxkryo
jackaroo
jackaroo

Reputation: 13

how to register graphx Edge class with Kryo?

I've been trying to register Edge class with Kryo but I'm always getting the following error.

java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.graphx.Edge\nNote: To register this class use: kryo.register(org.apache.spark.graphx.Edge.class);

what is wrong with following line?

sc.getConf.registerKryoClasses(Array(Class.forName("org.apache.spark.graphx.Edge")))

How should I do it?

Upvotes: 0

Views: 113

Answers (1)

kmh
kmh

Reputation: 1586

I've had trouble getting graphx classes registered. This finally works for me...

import org.apache.spark.graphx.GraphXUtils

val conf = new SparkConf().setAppName("yourAppName")
GraphXUtils.registerKryoClasses(conf)

Here's what's going on behind the scenes...

https://github.com/amplab/graphx/blob/master/graphx/src/main/scala/org/apache/spark/graphx/GraphKryoRegistrator.scala

In your case... I'm not sure why the following wouldn't work fine, since Edge is exposed...

conf.registerKryoClasses(Array(classOf[Edge]))

But I think there are private classes in graphx that aren't exposed through the spark API, at least I see them in the graphx repo, but not the spark.graphx repo. In my case, I couldn't get VertexAttributeBlock registered, until I used the GraphXUtils method.

Upvotes: 0

Related Questions