CODEWITHSUNDEEP
javalinuxanthadoop64-bit
zfz
zfz

Reputation: 1637

How to build native library for Hadoop 2

I've setup a cluster running Hadoop 2.1 beta on 64 bit linux. However, each time I run the hadoop command tools, a warning message pops out:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for 
your platform...
using builtin-java classes where applicable

Then I found out that it is lacking the native library for the the 64 bit linux. The official hadoop 2.1 tarball only provides the native library for 32 bit linux in /lib/native folder.

And I read the official document for hadoop native library, the guide says:

Once you installed the prerequisite packages use the standard hadoop 
build.xml file and pass along the compile.native flag (set to true) to 
build the native hadoop library:

$ant -Dcompile.native=true <target>

I search the hadoop folder, there is no file named build.xml. Haven't enough knowledge of java programming and hadoop, so I want to know how can I compile the native library for the 64 bit linux system? Thanks.

Upvotes: 2

Views: 6216

Answers (2)

Kshitiz Sharma
Kshitiz Sharma

Reputation: 18627

Download and install protobuf

wget http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz /tmp
tar -xvf protobuf-2.5.0.tar.gz
cd /tmp/protobuf-2.5.0
sudo ./configure
sudo ./make install 
sudo ldconfig

Install cmake

sudo apt-get install cmake

Build native libraries using maven

mvn compile -Pnative

If there are any errors run maven with -e -X switch which will output details debugging information. Look at what the error is and make suitable changes.

For example I got the following errors:

[ERROR] Could not find goal 'protoc' in plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT among available goals -> [Help 1]

Means you have incorrect protobuf version.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/tmp/hadoop-2.5.0-src/hadoop-common-project/hadoop-common/target/native"): error=2, No such file or directory

cmake is not installed.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive:
[ERROR] Exit code: 1 - /tmp/hadoop-2.5.0-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27: error: unexpected end tag: </ul>
[ERROR] * </ul>

Don't know what the issue is. Just skip the javadoc generation by passing the -Dmaven.javadoc.skip=true flag to maven.

Upvotes: 1

boneill42
boneill42

Reputation: 51

The build system has changed to maven. You can find the instructions for building here: https://svn.apache.org/repos/asf/hadoop/common/trunk/BUILDING.txt

Specifically, you can run this: mvn package -Pdist,native,docs -DskipTests -Dtar

(once you've install protobuf 2.5.0)

Upvotes: 4

Related Questions