CODEWITHSUNDEEP
scalaserializationapache-sparkkryo
user_s
user_s

Reputation: 1078

Scala IllegalArgumentException: can't serialize class

I have a very simple class that I'm trying to reduce using spark. For some reason it keep throwing exception can't serialize class. This is my class: 

@SerialVersionUID(1000L)
class TimeRange(val  start: Long, val end: Long) extends Serializable {

  def this(){
    this(0,0)
  }

  def mergeOverlapping(rangesSet : Set[TimeRange]) = {
    def minMax(t1: TimeRange, t2: TimeRange) : TimeRange = {
      new TimeRange(if(t1.start < t2.start) t1.start else t2.start, if(t1.end > t2.end) t1.end else t2.end)
    }
    (rangesSet ++ Set(this)).reduce(minMax)
  }


  def containsSlice(timeRange: TimeRange): Boolean ={
    (start < timeRange.start && end > timeRange.start) ||
      (start > timeRange.start && start < timeRange.end)
  }

  override def toString = {
    "("+ start + ", " + end + ")"
  }
}

I've tried it with @SerialVersionUID(2L) as well and with Kryo: 

conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.registerKryoClasses(Array(classOf[TimeRange], classOf[Set[TimeRange]]))

I'm using Scala 2.11 with Spark 1.6.1.

Edit: Using the same mapper and reducer but instead of using the class TimeRange using (Long, Long) works. 

 def mergeOverlapping(currRange : (Long, Long), rangesSet : Set[(Long, Long)]) = {
     def minMax(t1: (Long, Long), t2: (Long, Long)) : (Long, Long) = {
       (if(t1._1 < t2._1) t1._1 else t2._1, if(t1._2 > t2._2) t1._2 else t2._2)
     }
     (rangesSet ++ Set(currRange)).reduce(minMax)
   }

    def containsSlice(t1: (Long, Long), t2 : (Long, Long)): Boolean ={
      (t1._1 < t2._1 && t1._2 > t2._1) ||
        (t1._1 > t2._1 && t1._1 < t2._2)
    }

Upvotes: 1

Views: 252

Answers (1)

Jason CHAN
Jason CHAN

Reputation: 6815

I think this class can be serialized, but when you use it in Spark, you have to make sure all the variables in the closure set is serializable. I encountered this kind of problem when I was learning Spark, and most of the time it was caused by involving another variable that is not serializable into the closure (the code in map or reduce method). If you can show the code how you use this class, that may be helpful.

Upvotes: 1

Related Questions