Spark-Scala-Intellij java.lang.NoSuchMethodError

Question

I'm working with Apache Spark and Scala on Intellij. I have no experience with Maven. I am trying to put together a simple WordCount program and using JDK 11, Scala 2.12.12 and Spark 3.0.1. The project compiles fine but at the time of running, this is the error I get:-

Exception in thread "main" java.lang.NoSuchMethodError: 'void scala.util.matching.Regex.<init>(java.lang.String, scala.collection.Seq)'
    at scala.collection.immutable.StringLike.r(StringLike.scala:284)
    at scala.collection.immutable.StringLike.r$(StringLike.scala:284)
    at scala.collection.immutable.StringOps.r(StringOps.scala:33)
    at scala.collection.immutable.StringLike.r(StringLike.scala:273)
    at scala.collection.immutable.StringLike.r$(StringLike.scala:273)
    at scala.collection.immutable.StringOps.r(StringOps.scala:33)
    at org.apache.spark.util.Utils$.<init>(Utils.scala:104)
    at org.apache.spark.util.Utils$.<clinit>(Utils.scala)
    at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:70)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:59)
    at WordCount$.main(WordCount.scala:9)
    at WordCount.main(WordCount.scala)

I've checked the most common cause for this error. There is no provided in spark-core dependency, it's compile and the versions of Spark and Scala are recent.

This is the code:-

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

object WordCount {

  def main(args: Array[String]) {

    val conf = new SparkConf().setAppName("Spark Scala WordCount Example").setMaster("local[1]")

    val sc = new SparkContext(conf)

    var map = sc.textFile("/Users/<username>/Downloads/TestFile.csv").flatMap(line => line.split(",")).map(word => (word,1))

    var counts = map.reduceByKey(_ + _)

    counts.collect().foreach(println)

    sc.stop()
  }
} 

And this is the pom.xml I've used:-

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.org.example</groupId>
    <artifactId>Project</artifactId>
    <version>1.0-SNAPSHOT</version>

    <inceptionYear>2008</inceptionYear>
    <packaging>jar</packaging>
    <properties>
        <scala.version>2.12.12</scala.version>
        <spark.version>3.0.1</spark.version>
    </properties>

    <repositories>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <dependency>
            <groupId>org.specs</groupId>
            <artifactId>specs</artifactId>
            <version>1.2.5</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>${spark.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>${spark.version}</version>
            <scope>compile</scope>
        </dependency>

    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <resources><resource><directory>src/main/resources</directory></resource></resources>
        <plugins>
            
        </plugins>
    </build>
    <reporting>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
        </plugins>
    </reporting>

</project>

I tried running the same program with sbt before and it worked absolutely fine yet it always returns this error with maven.

Answer 1

I suspect this is an issue with the Scala SDK defined for your project.

When I copied your pom.xml to a fresh IDEA project, I received the `No Scala SDK Module" error. You can right click the project and select "Add framework support":

And then select Scala and the proper version. Once I did this and ran the main function, it ran properly.

If you already have a Scala version set, go to File -> Project Structure -> Global Libraries and make sure the proper Scala SDK is defined for the project: