CODEWITHSUNDEEP
linuxshellhadoophdfshadoop-yarn
user3813256
user3813256

Reputation:

connection exception when using hadoop2 (YARN)

I have setup Hadoop (YARN) on ubuntu. The resource manager appears to be running. When I run the hadoop fs -ls command, I receive the following error:

14/09/22 15:52:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: Call From ubuntu-8.abcd/xxx.xxx.xxx.xxxx to ubuntu-8.testMachine:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

I checked on the suggested URL in the error message but could not figure out how to resolve the issue. I ahve tried setting the external IP address (as opposed to localhost) in my core-site.xml file (in etc/hadoop) but that has not resolved the issue. IPv6 has been disabled on the box. I am running the process as hduser (which has read/write access to the directory). Any thoughts on fixing this? I am running this on a single node.

bashrc

#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HADOOP_INSTALL=/usr/local/hadoop/hadoop-2.5.1
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export HADOOP_YARN_HOME=$HADOOP_INSTALL  ##added because I was not sure about the line below
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
#HADOOP VARIABLES END

Upvotes: 2

Views: 577

Answers (2)

Roman Nikitchenko
Roman Nikitchenko

Reputation: 13046

Your issue is not related to YARN. It is limited by HDFS usage. Here is the question with similar situation - person who asked had 9000 port listening on external IP interface but configuration was pointing to localhost. I'd advise first check if somebody at all listens on port 9000 and on what interface. Looks like you have service listening on IP interface which differs from where you look for it. Looking at your logs your client is trying ubuntu-8.testMachine:9000. To what IP it is being resolved? If it is assigned in /etc/hosts to 127.0.0.1, you could have the situation as in question I have mentioned - client tries to access 127.0.0.1 but service is waiting on external IP. OK, you could have vice versa. Here is good default port mapping table for Hadoop services.

Indeed many similar cases have the same root - wrongly configured host interfaces. People often configure their workstation hostname and assign this hostname to localhost in /etc/hosts. More, they write first short name and only after this FQDN. But this means IP is resolved into short hostname but FQDN is resolved into IP (non-symmetric).

This in turn provokes number of situations where services are started on local 127.0.0.1 interface and people have serious connectivity issues (are you surprised? :-) ).

Right approach (at least I encourage it based on expirience):

  1. Assign at least one external interface that is visible to your cluster clients. If you have DHCP and don't want to have static IP, please bind your IP to MAC but move to 'constant' IP value.
  2. Write local hostname into /etc/hosts to match external interface. FQDN name first and then short.
  3. If you can, make your DNS resolver to resolve your FQDN into your IP. Don't care about short name.

Example, you have external IP interface 1.2.3.4 and FQDN (fully qualified domain name) set to myhost.com - in this case your /etc/hosts record MUST look like:

1.2.3.4 myhost.com myhost

And yes, it's better your DNS resolver knows about your name. Check both direct and reverse resolution with:

host myhost.com host 1.2.3.4

Yes, clustering is not so easy in term of networking administration ;-). Never has been and shall never be.

Upvotes: 2

Kaiser
Kaiser

Reputation: 21

Be sure you that you had started all the necesary, type start-all.sh, this command will start all the services needed for the connection to hadoop.

After that, you can type jps, with this command you can see all the services running under hadoop, and at the end, check the ports opened of these services with netstat -plnet | grep java.

Hope this solve your issue.

Upvotes: 1

Related Questions