CODEWITHSUNDEEP
apache-spark
beatrice
beatrice

Reputation: 4431

Spark InvalidClassException

I am try to send code to Spark master from intellij directly:

  def main(args: Array[String]) = {
    println("***hello spark-test***")
    val spark = SparkSession
      .builder()
      .master("spark://172.22.208.1:7077")
      .appName("Spark-Test Application")
      .getOrCreate()
    import spark.implicits._
    val rawData = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
    val rdd = spark.sparkContext.parallelize(rawData)
    val df = rdd.toDF()
    val result = df.filter($"value" % 2 === 1).count()
    println(s"***Result odd numbers count: $result ***")
    spark.stop()
  }

result:
the application log hangs at connecting to master:

22/12/29 12:38:55 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://172.22.208.1:7077...
22/12/29 12:38:55 INFO TransportClientFactory: Successfully created connection to /172.22.208.1:7077 after 38 ms (0 ms spent in bootstraps)
22/12/29 12:39:15 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://172.22.208.1:7077...

driver log:

22/12/29 12:39:35 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
java.io.InvalidClassException: scala.collection.immutable.ArraySeq; local class incompatible: stream classdesc serialVersionUID = -8615987390676041167, local class serialVersionUID = 2701977568115426262

spark version : spark-3.3.0-hadoop3-scala2.13
intellij's scala version: v2.13.5

However when I remove the line 'master("spark://172.22.208.1:7077")' and build a jar and send it via spark-submit it works fine.

Upvotes: 0

Views: 269

Answers (1)

Anthony Granger
Anthony Granger

Reputation: 814

Usually this error means that you are not running the same Spark version or the same Scala version.

You need to make sure that the code you are running on Intellij is running on the same Scala and Spark version than the one you are running on your cluster.

Since you mentionned you are running Scala 2.13 on IntelliJ, I guess that your local Spark version is not the same as the one you have on the cluster.

Upvotes: 1

Related Questions