HDFS pseudodistributed mode namenodes startup error

Question

I'm trying to start HDFS on Mac OS X (Java 7) in pseudodistributed mode. I've created a dir with config files in it, following the instructions found in various places (e.g., https://hadoop.apache.org/docs/r1.2.1/single_node_setup.html). I can ssh to localhost without needing a password. But when I try starting hdfs, I get the following:

$ start-dfs.sh --config ~/hadoop-pseudodistributed
2014-03-12 01:15:14.125 java[84567:1903] Unable to load realm info from SCDynamicStore
Starting namenodes on [2014-03-12 01:15:14,380 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
localhost]
2014-03-12: ssh: Could not resolve hostname 2014-03-12: nodename nor servname provided, or not known
Unable: ssh: Could not resolve hostname Unable: nodename nor servname provided, or not known
[main]: ssh: Could not resolve hostname [main]: nodename nor servname provided, or not known
WARN: ssh: Could not resolve hostname WARN: nodename nor servname provided, or not known
load: ssh: Could not resolve hostname load: nodename nor servname provided, or not known
-: ssh: Could not resolve hostname -: nodename nor servname provided, or not known
for: ssh: Could not resolve hostname for: nodename nor servname provided, or not known
native-hadoop: ssh: Could not resolve hostname native-hadoop: nodename nor servname provided, or not known
where: ssh: Could not resolve hostname where: nodename nor servname provided, or not known
builtin-java: ssh: Could not resolve hostname builtin-java: nodename nor servname provided, or not known
your: ssh: Could not resolve hostname your: nodename nor servname provided, or not known
applicable: ssh: Could not resolve hostname applicable: nodename nor servname provided, or not known
(NativeCodeLoader.java:<clinit>(62)): ssh: Could not resolve hostname (NativeCodeLoader.java:<clinit>(62)): nodename nor servname provided, or not known
using: ssh: Could not resolve hostname using: nodename nor servname provided, or not known
classes: ssh: Could not resolve hostname classes: nodename nor servname provided, or not known
platform...: ssh: Could not resolve hostname platform...: nodename nor servname provided, or not known
library: ssh: Could not resolve hostname library: nodename nor servname provided, or not known
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.2.0/libexec/logs/hadoop-terry-namenode-Terrys-MacBook-Pro.local.out
01:15:14,380: ssh: Could not resolve hostname 01:15:14,380: nodename nor servname provided, or not known
to: ssh: connect to host to port 22: Connection refused
localhost: 2014-03-12 01:15:15,150 INFO  [main] namenode.NameNode (StringUtils.java:startupShutdownMessage(601)) - STARTUP_MSG:

There's more output (I get similar complaints as it tries to start secondary namenodes), but the above is clearly undesirable and I'd obviously like to fix it.

It looks like the script is running something to get a list of namenodes and that thing is dumping an error (to stdout or stderr) which is being captured and used as a list of nodes.

I tried unsuccessfully to clear the "Unable to load realm info from SCDynamicStore" error by adding to hadoop-env.sh (as suggested elsewhere on stackoverflow). But that seems to go outside the documented setup steps, which do not include putting a copy of hadoop-env.sh into my config dir.

I guess this should be easy, but it's late here and I'm tired :-( Any help would be appreciated.

Thanks!

Answer 1

I have a really satisfying workaround for this issue when trying to run Hadoop in pseudo-distributed mode on MacOS.

# use hadoop-daemon.sh instead of start-dfs.sh
# because start-dfs.sh relies on native libs not present in MacOS
/platform/hadoop/sbin/hadoop-daemon.sh start namenode
/platform/hadoop/sbin/hadoop-daemon.sh start secondarynamenode
/platform/hadoop/sbin/hadoop-daemon.sh start datanode

# use hadoop-daemon.sh instead of stop-dfs.sh
# because stop-dfs.sh relies on native libs not present in MacOS
/platform/hadoop/sbin/hadoop-daemon.sh stop datanode
/platform/hadoop/sbin/hadoop-daemon.sh stop secondarynamenode
/platform/hadoop/sbin/hadoop-daemon.sh stop namenode

I know this is 3 years old but hopefully this saves someone else the hassle and wasted effort I went through - I spent waaay too much time trying to build from source to get the native libs that you need to point to with the changes to hadoop-env.sh to make start-dfs.sh and stop-dfs.sh work, before seeing this question, reading the scripts to see what they were calling ($HADOOP_PREFIX/bin/hdfs getconf -namenodes) and realizing that since I'm only interested in pseudo-distributed mode, with one node of each type, I can just say f*** the 'convenience' script and use hadoop-daemon.sh to start and stop them myself.

And I personally don't need this, but you could even overwrite the contents of start-dfs.sh and stop-dfs.sh with this workaround if you're doing something where those scripts get called by something else.

Answer 2

The terminal session on the mac is unable to make connections to localhost if you had been running the applications earlier with hadoop default configuration folder.

Please close the current terminal and run the applications in a new terminal.

Answer 3

2014-03-12 01:15:14.125 java[84567:1903] Unable to load realm info from SCDynamicStore Starting namenodes on [2014-03-12 01:15:14,380 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable localhost]

Above line in your error says "Unable to load native-hadoop library". This is because native library is compiled(built) on 32 bit and you might be running it on 64 bit. I think these lines should be WARNING. Still if it is saying error, you should follow this link.

Further:

2014-03-12: ssh: Could not resolve hostname 2014-03-12: nodename nor servname provided, or not known

I dont know about Mac OS but i will try to give analogy with respect to Ubuntu.Below lines i have added in .bashrc of ubuntu. Means i am giving path of native libraries in .bashrc so that OS knows about it.

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"

In similar fashion you have to set path to native libraries in your OS. Hoping this will solve your "Could not resolve hostname " problem(I came across similar problem but in Ubuntu, and it worked)