CODEWITHSUNDEEP
scalaapache-spark
Yves
Yves

Reputation: 91

why scala compiler failed to infer type when i write PairDStreamFunctions.reduceByKey

I wonder why scala compiler couldn't infer a type of my function parameter when i using PairDStreamFunctions.reduceByKey,here is code:

val ssc = new StreamingContext(conf, Seconds(10))
ssc.checkpoint(".checkpoint")
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" "))
val wordCounts = words
  .map((_, 1))
  .reduceByKey((x: Int, y: Int) => x + y, 4)  //here i must specify the type Int,and this format can't work : reduceByKey((x, y) => x + y, 4)

here i must specify the type Int of my function parameter like reduceByKey((x: Int, y: Int) => x + y, 4) when i use PairDStreamFunctions.reduceByKey ,and this format couldn't work : reduceByKey((x, y) => x + y, 4).

On the other hand,when i use PairRDDFunctions.reduceByKey api,it can infer the type,here is the code:

val conf = new SparkConf()
val sc = new SparkContext(conf)
val rdd = sc.parallelize(List(
  "hi what"
  , "show you"
  , "matter how"
))
rdd.flatMap(_.split(" "))
  .map((_, 1))
  .reduceByKey((x, y) => x + y, 4)//in this code,scala compiler could infer the type of my function parameter (x,y) => x+y

When i use PairRDDFunctions.reduceByKey, reduceByKey((x, y) => x + y, 4) could work. I really don't understand what makes it different?

Upvotes: 3

Views: 64

Answers (1)

Justin Pihony
Justin Pihony

Reputation: 67075

This is happening because the PairRDDFunctions method only has one overload of def reduceByKey(func: (V, V) ⇒ V, [SOMETHING]) whereas PairDStreamFunctions has two:

def reduceByKey(reduceFunc: (V, V) ⇒ V, numPartitions: Int)
def reduceByKey(reduceFunc: (V, V) ⇒ V, partitioner: Partitioner)

So, although the partitioner variant should be thrown out as a possibility, it still comes in and confuses the compiler. You can see this by explicitly naming it:

.reduceByKey((x, y) => x + y,partitioner = 4)

I'm not sure where this falls in the compiler definitions, but it is clearly exhibited as the reason per the above.

Upvotes: 1

Related Questions