CODEWITHSUNDEEP
scalaserializationhadoopapache-spark
mbaryu
mbaryu

Reputation: 189

How do I make a Scala class Hadoop Writable?

I have a Scala class:

class Features (
  val count: Int = 1,
  val firstTime: Long = Long.MaxValue,
  val lastTime: Long = Long.MinValue)

which is held in a Spark pair RDD with String as the key:

val features: org.apache.spark.rdd.RDD[(String, features.Features)]

and I would like to save the contents with something like:

features.saveAsSequenceFile(path)

However the Features class does not implement org.apache.hadoop.io.Writable and I don't know how to implement this. Without it, the error I get is:

No implicit view available from features.Features => org.apache.hadoop.io.Writable.

How do I make Features Hadoop Writable?

Upvotes: 1

Views: 1044

Answers (1)

1esha
1esha

Reputation: 1722

You should implement Writable interface in your classes. There are two general ways:

  1. Implement writable in you case class

  2. Implement type class and provide implicit conversion to it

Upvotes: 1

Related Questions