CODEWITHSUNDEEP
scalatypesapache-sparkmismatch
3xCh1_23
3xCh1_23

Reputation: 1499

Apache Spark type mismatch of the same type (String)

EDIT: Answer: It was a JAR file that created a conflict! The related post is: Must include log4J, but it is causing errors in Apache Spark shell. How to avoid errors?

Doing the following:

val numOfProcessors:Int = 2
val filePath:java.lang.String = "s3n://somefile.csv"
var rdd:org.apache.spark.rdd.RDD[java.lang.String] = sc.textFile(filePath, numOfProcessors)

I get

    error: type mismatch;
 found   : org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
 required: org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
       var rdd:org.apache.spark.rdd.RDD[java.lang.String] = sc.textFile(filePath, numOfProcessors)

EDIT: Second case

val numOfProcessors = 2
val filePath = "s3n://somefile.csv"
var rdd = sc.textFile(filePath, numOfProcessors) //OK!

def doStuff(rdd: RDD[String]): RDD[String] = {rdd}

doStuff(rdd)

I get:

error: type mismatch;
 found   : org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
 required: org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
              doStuff(rdd)
                      ^

No comment...

Any ideas why I get this error ?

Upvotes: 2

Views: 1091

Answers (1)

3xCh1_23
3xCh1_23

Reputation: 1499

The problem was a JAR file that created a conflict.

Upvotes: 2

Related Questions