Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

Question

trying to run MR program version(2.7) in windows 7 64 bit in eclipse while running the above exception occurring . I verified that using 64 bit 1.8 java version and observed that all the hadoop daemons are running.

Any suggestions highly appreciated

Answer 1

winutils.exe requires Visual C++ Redistributable for Visual Studio 2015. You can check it by running winutils.exe from command line. Also in my case also only the latest version of winutils worked: hadoop-3.4.0-win10-x64 from https://github.com/kontext-tech/winutils Older version work only with single file, not whole directory.

Answer 2

Hadoop requires native access to the file system on Windows. This is provided by Winutils, a kind of native access bridge.

1. Download the winutils.exe and hadoop.dll from https://github.com/steveloughran/winutils or https://github.com/kontext-tech/winutils

2. Put winutils.exe in a bin directory somewhere on your disk (e.g. C:\winutils\bin\winutils.exe)

3. Put hadoop.dll anywhere on you disk (e.g. C:\winutils\bin\hadoop.dll)

4. Add the following code to your application:

    System.setProperty ("hadoop.home.dir", "C:/winutils/" ); 
    System.load ("C:/winutils/bin/hadoop.dll");
   

This way, it minimizes the impact on your system: you don't have to modify your environment variables or to put the DLL into the Windows\System32 directory.

Answer 3

If the issue still persists after adding the correct version of hadoop winutils + hadoop.dll, check if you have installed JRE or JDK.

Hadoop is written in Java. So it would require JRE (Java Runtime Environment) at the very least with the supported version. But JRE (Java Runtime Environment) is insufficient. For running Map-Reduce jobs, Hadoop essentially requires JDK (Java Development Kit). This is why after installation (with JRE), when you run the command pyspark in the terminal, you get this Unable to load native-hadoop library for your platform warning. For me, switching to JDK resolved these issues.

Here's a thread on JDK requirement for Hadoop: https://www.quora.com/Why-we-need-JDK-for-Hadoop-installation

Answer 4

Whereas all the above recommendations failed to resolve this error for me. Setting the environment variable HADOOP_BIN_PATH resolved the error for me. Also another related error where the word access0 in the error string is replaced with createDirectoryWithMode0.

If you look in the hadoop/bin folder and inspect hdfs.cmd and mapred.cmd it is clear that the HADOOP_BIN_PATH environment variable is expected. For example this code:

if not defined HADOOP_BIN_PATH ( 
  set HADOOP_BIN_PATH=%~dp0
)

%~dp0 should expand to the folder containing the cmd file. However, you could imagine other components of hadoop such as the daemons that do not start from the command line which may also expect this environment variable to be set.

Set HADOOP_BIN_PATH to the bin folder under your hadoop directory.

If you set a user environment variable you will need to restart the process running your application. If you set a system environment variable you will need to restart Windows to see the effect.

Answer 5

After putting hadoop.dll and winutils in hadoop/bin folder and adding the folder of hadoop to PATH, we also need to put hadoop.dll into the C:\Windows\System32 folder

Answer 6

In my case I was having this issue when running unit tests on local machine after upgrading dependencies to CDH6. I already had HADOOP_HOME and PATH variables configured properly but I had to copy the hadoop.dll to C:\Windows\System32 as suggested in the other answer.

Answer 7

For me, I have to download the winutils from https://github.com/kontext-tech/winutils as it has the latest version 3.3.

It is important to make sure the version matches to the Hadoop version you downloaded (with or without Spark) otherwise, you can find some weird error messages.

Both hadoop.dll and winutils.exe are fine to be at the same folder C:/hadoop/bin. I didn't copy either to system folder and it works.

Note: I followed this except the download pages of winutils tool.

Answer 8

I'm using Spark version 3.1.2 and hadoop 3.2 (spark-3.1.2-bin-hadoop3.2).

I solved this by just downloading the hadoop.dll file from a Github page = https://github.com/cdarlint/winutils and saving it to the bin folder within my spark folder. Then spark-submit went smooth.

Answer 9

In my case (pyspark = 3.3.1, Spark version = 3.3.1, Hadoop version = 3.3.2) I set env vars by python code

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['HADOOP_HOME'] = "C:\\Program Files\\Hadoop\\"

added from https://github.com/kontext-tech/winutils to bin folder last version of hadoop files hadoop-3.4.0-win10-x64 and added hadoop.dll to C:\Windows\System32

Answer 10

After downloading and configuring hadoop.dll and wintuils.exe as previous answer, you need to "restart the windows" to make it works.

Answer 11

This is what worked for me: Download the latest winutils https://github.com/kontext-tech/winutils Or check your spark Release text, it shows the cer of Hadoop it is using

Steps

1. Dowload repo

2. Create a folder named hadoop anywhere (e.g. desktop/hadoop)

3. Paste the bin into that folder (you will then have hadoop/bin)

4. copy hadoop.dll to windows/system32

5. Set system environment:

   set HADOOP_HOME=c:/desktop/hadoop 
   set PATH=%PATH%;%HADOOP_HOME%/bin;
   

Answer 12

I already had %HADOOP_HOME%/bin in my PATH. Adding hadoop.dll in Hadoop/bin directory made it work again.

Answer 13

Adding hadoop.dll and WinUntils.exe fixed the error, support for latest versions can be found here

Answer 14

This might be old but if its still not working for someone, Step 1- Double click on winutils.exe. If it shows some dll file is missing, download that .dll file and place that at appropriate place.

In my case, msvcr100.dll wasmissing and I had to install Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package to make it work. All the best

Answer 15

After trying all the above, things worked after putting hadoop.dll to windows/System32

Answer 16

The version mismacth is main cause for this issue. Follow complete hadoop version with java library will solve the issue and if you still face issue and working on hadoop 3.1.x version use this library to download bin

https://github.com/s911415/apache-hadoop-3.1.0-winutils/tree/master/bin

Answer 17

Yes this issues arose when I was using the PIGUNITS for automation of PIGSCRIPTS. Two things in sequence need to be done:

1. Copy both the files as mentioned about in a location and add it to the environment variables under PATH.

2. To reflect the change what you have just done, you have to restart your machine to load the file.

Under JUNIT I was getting this error which would help others as well:

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias XXXXX. Backend error : java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING at org.apache.pig.PigServer.openIterator(PigServer.java:925)

Answer 18

For me this issue was resolved by downloading the winutils.exe & hadoop.dll from https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin and putting those in hadoop/bin folder

Answer 19

In addition to other solutions, Please download winutil.exe and hadoop.dll and add to $HADOOP_HOME/bin. It works for me.

https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin

Note: I'm using hadoop-2.7.3 version

Answer 20

In Intellij under Run/Debug Configurations, open the application you are trying to run, Under configurations tab, specify the exact working Directory.having the variable to represent the working directory also creates this problem. When I changed the Working Directory under configurations, it started working again.

Answer 21

I already had %HADOOP_HOME%/bin in my PATH and my code had previously run without errors. Restarting my machine made it work again.

Answer 22

This issue occurred to me and the cause was I forgot to append %HADOOP_HOME%/bin to PATH in my environment variables.