CODEWITHSUNDEEP
hadoophadoop2bigdata
Sujoy
Sujoy

Reputation: 117

Should I have to run history server in all nodes to get job history in Hadoop Cluster WebUI

I am facing one issue in Hadoop cluster. I have a Hadoop cluster with 5 datanodes and one edge/gateway node. My issue is that I had to start the history server in each of those nodes (1 namenode and 5 datanodes) to get any job history from hadoop webUI for any submitted job. I have added mapreduce.jobhistory.address and mapreduce.jobhistory.webapp.address in mapred-site.xml But it's not working properly I guess. If I start the history server in name node or any other node only , Hadoop Cluster Web-UI is unable to show me the job history and ends up with some error.

My Mapred-site XML

<configuration>
<property>
    <name>mapred.job.tracker</name>
    <value>hadoopmaster:8021</value>
</property>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

<property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoopmaster:10020</value>
    </property>

<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoopmaster:19888</value>
    </property>

</configuration>

For the time being as a workaround I start the history server in each node (namenode and all data node) manually. But think this is not right way.

Now I have 5 data node only so its still feasible to start history server in each and every node manually , but if case of multiple nodes(say 100/200) it will not be feasible any more to start history server in every node. There should be some standard solution for this issue...

Please help me out if anyone knows how to resolve this issue. Thanks in advance….

Upvotes: 2

Views: 974

Answers (1)

Sujoy
Sujoy

Reputation: 117

Finally I am able to solve the issue. Actually in case of mapreduce.jobhistory.address , it will history server is running in one node only (jps). It's working properly now...

Upvotes: 1

Related Questions