CODEWITHSUNDEEP
scaladistributedapache-sparkspark-streaming
user2895478
user2895478

Reputation: 383

Spark Streaming Window Operation

The following is simple code to get the word count over a window size of 30 seconds and slide size of 10 seconds.

import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.api.java.function._
import org.apache.spark.streaming.api._
import org.apache.spark.storage.StorageLevel

val ssc = new StreamingContext(sc, Seconds(5))

// read from text file
val lines0 = ssc.textFileStream("test")
val words0 = lines0.flatMap(_.split(" "))

// read from socket
val lines1 = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER)
val words1 = lines1.flatMap(_.split(" "))

val words = words0.union(words1)
val wordCounts = words.map((_, 1)).reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))

wordCounts.print()
ssc.checkpoint(".")
ssc.start()
ssc.awaitTermination()

However, I am getting error from this line: 

val wordCounts = words.map((_, 1)).reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))

. Especially, from _ + _. The error is 

51: error: missing parameter type for expanded function ((x$2, x$3) => x$2.$plus(x$3))

Could anybody tell me what the problem is? Thanks!

Upvotes: 6

Views: 4562

Answers (1)

aaronman
aaronman

Reputation: 18750

This is extremely easy to fix, just be explicit about the types.
val wordCounts = words.map((_, 1)).reduceByKeyAndWindow((a:Int,b:Int)=>a+b, Seconds(30), Seconds(10))

The reason scala can't infer the type in this case is explained in this answer

Upvotes: 10

Related Questions