CODEWITHSUNDEEP
hadoop
EnDelt64
EnDelt64

Reputation: 1360

I can't access to Hadoop Web Interface (DataNode, ResourceManager)

I've installed Hadoop 3.1.1 with pseudo-distributed mode. I tried to access Hadoop Web Interface, and NameNode(Server's public IP:9870) and JobHistoryServer(public IP:19888) UI are opened well but DataNode(public IP:9864), ResourceManager(public IP:8088) UI are blocked.

However, when I put the command jps, DataNode and ResourceManager is still working. Also, there was nothing special error message in log file.

enter image description here

I want to know what is the problem.

masters:

localhost

slaves:

localhost

hdfs-site.xml:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>(Hadoop Home Dir)/hdata/dfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>(Hadoop Home Dir)/hdata/dfs/datanode</value>
  </property>
  <property>
    <name>dfs.datanode.http.address</name>
    <value>localhost:9864</value>
  </property>
</configuration>

core-site.xml:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>(Hadoop Home Dir)/hdata</value>
  </property>
</configuration>

mapred-site.xml:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml:

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
  </property>
  <property>
    <name>yarn.web-proxy.address</name>
    <value>localhost:8089</value>
  </property>
</configuration>

ResourceManager logs:

2018-09-23 17:09:07,192 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting ResourceManager
STARTUP_MSG:   host = ubuntu-1cpu-40gb_ssd-2gb_ram-2tb_bw/127.0.1.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 3.1.1

Upvotes: 2

Views: 8126

Answers (1)

EnDelt64
EnDelt64

Reputation: 1360

I've modified config files like following, and now 6 processes and web interface are working well.

/etc/hosts (Local PC/WSL)

127.0.0.1                localhost
(Server's external IP)   (Server's hostname)

127.0.1.1 is need to be deleted.

masters:

(Server's external IP)

slaves:

(Server's external IP)

workers:

(Server's external IP)

hdfs-site.xml:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>(Hadoop Home Dir)/hdata/dfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>(Hadoop Home Dir)/hdata/dfs/datanode</value>
  </property>
</configuration>

core-site.xml:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>(Hadoop Home Dir)/hdata</value>
  </property>
</configuration>

mapred-site.xml:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml:

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

Upvotes: 1

Related Questions