CODEWITHSUNDEEP
scalaapache-sparkrddimplicit-conversioncompanion-object
Anurag Wagh
Anurag Wagh

Reputation: 1086

Scala Implicit Conversion for companion object of extended class

I am trying to create a customRDD in Java.

RDD converts RDD[(K,V)] to PairRDDFunctions[K,V] using Scala implicit function rddToPairRDDFunctions() defined in object RDD.

I am trying to do the same with my CustomJavaRDD which extends CustomRDD which extends RDD.

Now it should call implicit function rddToCustomJavaRDDFunctions() whenever it encounters CustomJavaRDD[(K,V)], but for some reason it still goes to rddToPairRDDFunctions().

What am I doing wrong?

RDD.scala

class RDD[T]
object RDD {
  implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
      (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null):
      PairRDDFunctions[K, V] = {
    new PairRDDFunctions(rdd)
  }
}

CustomRDD.scala

abstract class CustomRDD[T] extends RDD[T]
object CustomRDD {
  implicit def rddToCustomJavaRDDFunctions[K,V](rdd: CustomJavaRDD[(K,V)]): 
      PairCustomJavaRDDFunction[K,V] = {
    new PairCustomJavaRDDFunctions[K,V](rdd)
  }
}

PairCustomJavaRDDFunctions.scala

class PairCustomJavaRDDFunctions[K: ClassTag, V: ClassTag](self: CustomRDD[(K, V)])
                        (implicit ord: Ordering[K] = null) {
  def collectAsMap() = ???
}

There is no error; the program compiles successfully, but let's say I have data: RDD which is an instance of CustomJavaRDD.

data.collectAsMap()

At the runtime it converts data into PairRDDFunctions; i.e. it makes implicit call to rddToPairRDDFunctions defined in RDD.scala.

But it should make call to rddToCustomJavaRDDFunctions defined in CustomRDD.scala and convert it into PairCustomJavaRDDFunctions.

Upvotes: 0

Views: 498

Answers (1)

HTNW
HTNW

Reputation: 29193

But it should make call to rddToCustomJavaRDDFunctions defined in CustomRDD.scala and convert it into PairCustomJavaRDDFunctions

No, Scala simply does not work this way. What you want, overriding an implicit conversion depending on the runtime type of an object, is simply not possible (without pre-existing machinery on both the library's part and yours).

Implicits are a strictly compile-time feature. When the compiler sees you using an RDD as if it were a PairRDDFunctions, it splices in a call to RDD.rddToPairRDDFunctions, as if you wrote it yourself. Then, when the code is translated to bytecode, that call has already been baked in and nothing can change it. There is no dynamic dispatch for this, it's all static. The only situation where rddToCustomJavaRDDFunctions will be called is when the static type of the expression in question is already CustomJavaRDD.

Really, this should not be necessary. Implicit conversions are really no more than glorified helper methods that save you keystrokes. (Implicit parameters, now those are interesting. ;) ) There should be no need to override them because the helper methods should already be polymorphic and work whether you have RDD, CustomRDD, or `RDD that travels through time to compute things faster`.

Of course, you can still do it, but it will only actually do anything under the above conditions, and that is probably not very likely, making the whole thing rather pointless.

Upvotes: 1

Related Questions