CODEWITHSUNDEEP
scalaapache-spark
Karima Rafes
Karima Rafes

Reputation: 1108

Convert RDD[(Int,Int)] to PairRDD in scala

What is the problem with this example ?

val f = sc.parallelize(Array((1,1),(1,2)))
val p = new org.apache.spark.rdd.PairRDDFunctions[Int,Int](f)

Name: Compile Error
Message:  error: type mismatch;
 found   : org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[(Int, Int)]
 required: org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[(Int, Int)]
       val p = new org.apache.spark.rdd.PairRDDFunctions[Int,Int](f)
                                                                  ^

Upvotes: 1

Views: 333

Answers (1)

gil.fernandes
gil.fernandes

Reputation: 14621

Your code seems to work fine on Spark 2.2.0.

This is the transcript of the console commands in Spark version 2.2.0:

scala> val f = sc.parallelize(Array((1,1),(1,2)))
f: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> val p = new org.apache.spark.rdd.PairRDDFunctions[Int,Int](f)
p: org.apache.spark.rdd.PairRDDFunctions[Int,Int] = org.apache.spark.rdd.PairRDDFunctions@6e1d939e

scala> p
res0: org.apache.spark.rdd.PairRDDFunctions[Int,Int] = org.apache.spark.rdd.PairRDDFunctions@6e1d939e

scala> f
res1: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[0] at parallelize at <console>:24

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_131)

This seems to be a bug in an older version to me.

Upvotes: 1

Related Questions