CODEWITHSUNDEEP
scalaapache-sparkrdd
blue-sky
blue-sky

Reputation: 53806

reduceByKey method not being found in Scala Spark

Attempting to run http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala from source.

This line:

val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)

is throwing error 

value reduceByKey is not a member of org.apache.spark.rdd.RDD[(String, Int)]
  val wordCounts = logData.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)

logData.flatMap(line => line.split(" ")).map(word => (word, 1)) returns a MappedRDD but I cannot find this type in http://spark.apache.org/docs/0.9.1/api/core/index.html#org.apache.spark.rdd.RDD

I'm running this code from Spark source so could be a classpath problem ? But required dependencies are on my classpath.

Upvotes: 25

Views: 13450

Answers (3)

Hongyang
Hongyang

Reputation: 79

Actually, you can find it in PairRDDFunctions class. PairRDDFunctions is a class contains extra functions available on RDDs of (key, value) pairs through an implicit conversion.

https://spark.apache.org/docs/2.1.0/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions

Upvotes: 0

Ws576
Ws576

Reputation: 1

If you use maven on ScalaIDE I just solved the problem by updating the dependency from spark-streaming version 1.2 to version 1.3.

Upvotes: 0

maasg
maasg

Reputation: 37435

You should import the implicit conversions from SparkContext:

import org.apache.spark.SparkContext._

They use the 'pimp up my library' pattern to add methods to RDD's of specific types. If curious, see SparkContext:1296

Upvotes: 37

Related Questions