CODEWITHSUNDEEP
scalahadoopapache-spark
Shashi
Shashi

Reputation: 2714

Creating Map values in Spark using Scala

I am new to spark-scala development. I am trying to create a map values in spark using scala but getting type mismatch error.

scala> val nums = sc.parallelize(Map("red" -> "#FF0000","azure" -> "#F0FFFF","peru" -> "#CD853F"))
<console>:21: error: type mismatch;
 found   : scala.collection.immutable.Map[String,String]
 required: Seq[?]
Error occurred in an application involving default arguments.
       val nums = sc.parallelize(Map("red" -> "#FF0000","azure" -> "#F0FFFF","peru" -> "#CD853F"))

How should I do this?

Upvotes: 0

Views: 3910

Answers (1)

zero323
zero323

Reputation: 330413

SparkContext.parallelize transforms from Seq[T] to RDD[T]. If you want to create RDD[(String, String)] where each element is an individual key-value pair from the original Map use:

import org.apache.spark.rdd.RDD

val m = Map("red" -> "#FF0000","azure" -> "#F0FFFF","peru" -> "#CD853F")
val rdd: RDD[(String, String)] = sc.parallelize(m.toSeq)

If you want RDD[Map[String,String]] (not that it makes any sense with a single element) use:

val rdd: RDD[Map[String,String]] = sc.parallelize(Seq(m))

Upvotes: 4

Related Questions