CODEWITHSUNDEEP
apache-spark
MystCabe
MystCabe

Reputation: 57

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z Spark 3.3.1

Trying to create a table in Apache Spark gives the following error.

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

I have tried following the same steps as described here Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z, but with no luck.

I have downloaded Apache Spark 3.3.1 (spark-3.3.1-bin-hadoop3.tgz version) and the hadoop.dll file as well as the winutils.exe file from the 3.3.1/bin from https://github.com/kontext-tech/winutils.

I have added the environment variables. I have added the files to the system32 folder.

I have also tried it with Apache Spark 3.3.0.

But I still get the same error message. Not sure what else to do. Can there be some security setting that is blocking the files from working? I'm using a work computer after all.

EDIT: Some clarification

  1. I downloaded spark 3.3.1 (spark-3.3.1-bin-hadoop3.tgz) from the spark website. https://spark.apache.org/downloads.html
  2. I extracted the files into my C:\Spark folder.
  3. I downloaded the hadoop.dll file as well as the winutils.exe file from the 3.3.1/bin from https://github.com/kontext-tech/winutils I put these files into my C:\Hadoop\bin folder as well as the C:\Windows\system32 folder.
  4. I added the environmental variables HADOOP_HOME = C:\Hadoop and SPARK_HOME = C:\Spark as described in STEP 7 in https://phoenixnap.com/kb/install-spark-on-windows-10. I also added these variables to the path, i.e %SPARK_HOME%\bin and %HADOOP_HOME%\bin
  5. Restarted my computer.
'name := "Simple Project"

version := "1.0"

scalaVersion := "2.12.15"

libraryDependencies ++= Seq("org.apache.spark" %% "spark-sql" % "3.3.1", "io.delta" %% "delta-core" % "2.1.1")

Upvotes: 1

Views: 2625

Answers (1)

Koedlt
Koedlt

Reputation: 5973

Looks like you're doing the correct things! I expect there is something small wrong somewhere, so these are things you can look at:

  • Verify whether your JVM is a 32 or 64 bit JVM. You can do this as explained here. From the link you added it seems like these were pre-compiled for a 64 bit JVM. Maybe you are on a 32 bit one?
  • If it's not the previous point, I expect the error to be somewhere in your environment variables. Carefully study them, looking for unwated space characters, or wrong delimiters, or something else like that.

Hope this helps!

Upvotes: 2

Related Questions