Build Apache Hadoop from source on OS X in 2017

Question

Thus far I've forked the Git repo:

https://github.com/apache/hadoop

I've been looking around for information on how to build from source, I've tried this command:

mvn package -Pdist -Dtar -DskipTests

which resulted in the following error:

It doesn't seem that there's good documentation about this, does anyone know the right procedure?

---

UPDATE:

Also tried in this way:

According to this file, you need to get get protoc to work for the package to build

brew link --force --overwrite protobuf250

The macOS dependencies are:

$ brew install cmake
$ brew install zlib
$ brew install protobuf
$ brew install snappy

but to no avail:

By the way, the official docs are totally unhelpful- they essentially say "use a Docker container"

Answer 1

Older brew versions don't seem to exist anymore for protobuf 2.5.0. I found it worked by downloading this:

https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.bz2

then running the the regular install after you expand that file:

cd protobuf-2.5.0  
./configure
make
make check
sudo make install

Answer 2

Using Homebrew on OSX:

Install the required dependencies:

brew install protobuf@2.5 gcc autoconf automake libtool cmake snappy gzip bzip2 zlib openssl

Symlink protoc:

ln -s /usr/local/Cellar/protobuf@2.5/2.5.0/bin/protoc /usr/local/bin/protoc

Verify the version (libprotoc 2.5.0):

protoc --version

Export build flags:

export OPENSSL_ROOT_DIR="/usr/local/opt/openssl"
export LDFLAGS="-L${OPENSSL_ROOT_DIR}/lib"
export CPPFLAGS="-I${OPENSSL_ROOT_DIR}/include"
export PKG_CONFIG_PATH="${OPENSSL_ROOT_DIR}/lib/pkgconfig"
export OPENSSL_INCLUDE_DIR="${OPENSSL_ROOT_DIR}/include"

Check Hadoop version:

hadoop version

Fetch the Hadoop source for the version returned above and build:

wget https://archive.apache.org/dist/hadoop/core/hadoop-3.1.0/hadoop-3.1.0-src.tar.gz
tar zxvf hadoop-3.1.0-src.tar.gz
cd hadoop-3.1.0-src
mvn package -Dmaven.javadoc.skip=true -Pdist,native -DskipTests -Dtar

Copy the native libs to your Homebrew installation:

cp -R hadoop-dist/target/hadoop-3.1.0/lib/ /usr/local/Cellar/hadoop/3.1.0/lib/

Update hadoop-env.sh:

vi /usr/local/Cellar/hadoop/3.1.0/libexec/etc/hadoop/hadoop-env.sh

Amend HADOOP_OPTS (may be commented out):

export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/Cellar/hadoop/3.1.0/lib/native"

Restart Hadoop and run the following to verify:

hadoop checknative -a

Based on instructions from https://medium.com/@faizanahemad/hadoop-native-libraries-installation-on-mac-osx-d8338a6923db

Answer 3

Recently I've come up against this problem, too. After some googling, I found this issue involves some tricky thing between third-party TLS library(openssl) and the native one in macOS. Although I haven't figured out what's exactly going on here.

My solution:

Supposed you have already installed openssl through homebrew. Set these in command line or in ~/.bash_profile(and re-source it).

export OPENSSL_ROOT_DIR="/usr/local/opt/openssl"
export LDFLAGS="-L${OPENSSL_ROOT_DIR}/lib"
export CPPFLAGS="-I${OPENSSL_ROOT_DIR}/include"
export PKG_CONFIG_PATH="${OPENSSL_ROOT_DIR}/lib/pkgconfig"
export OPENSSL_INCLUDE_DIR="${OPENSSL_ROOT_DIR}/include"

Run the maven command again, you should be good to go.

Answer 4

mvn clean install -DskipTests

mvn package -Pdist -Pnative -Dtar -DskipTests