CODEWITHSUNDEEP
javascalaapache-sparkhadoop
Saurabh Soni
Saurabh Soni

Reputation: 21

Error: A JNI error has occurred, please check your installation and try again in IntelliJ IDEA for Scala-Spark Program using SBT

import org.apache.spark.sql.SparkSession

object RDDBroadcast extends App {

  val spark = SparkSession.builder()
    .appName("SparkByExamples.com")
    .master("local")
    .getOrCreate()

  val states = Map(("NY","New York"),("CA","California"),("FL","Florida"))
  val countries = Map(("USA","United States of America"),("IN","India"))

  val broadcastStates = spark.sparkContext.broadcast(states)
  val broadcastCountries = spark.sparkContext.broadcast(countries)

  val data = Seq(("James","Smith","USA","CA"),
    ("Michael","Rose","USA","NY"),
    ("Robert","Williams","USA","CA"),
    ("Maria","Jones","USA","FL")
  )

  val rdd = spark.sparkContext.parallelize(data)

  val rdd2 = rdd.map(f=>{
    val country = f._3
    val state = f._4
    val fullCountry = broadcastCountries.value(country)
    val fullState = broadcastStates.value(state)
    (f._1,f._2,fullCountry,fullState)
  })

  println(rdd2.collect().mkString("\n"))

}

The above is the spark-scala code to get country and state names. While compiling the code in InteliJ IDEA, I'm getting Error as below:

*Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 7 more*

I've tried multiple solutions like checking the java and javac version but the versions are in sync. Click here to view the image.

I've also checked the java version in the project File->project Structure...-> Module and compared it with Run->Edit Configurations which is also matching.

Build.sbt code: Click here to view the sbt code

I'm using VMware workstation 16 Player on which inteliJ is installed in linux OS. java version is 1.8.0_301

Click Here to view the InteliJ IDE in Linux Image

Upvotes: 1

Views: 2109

Answers (1)

ahmetsaim
ahmetsaim

Reputation: 103

I have faced the same issue, resolved it by setting Run/Debug Configurations -> Modify Options -> Java -> Add dependencies with provided scope to classpath to YES

Upvotes: 7

Related Questions