CODEWITHSUNDEEP
hadoopmapreduce
Jon
Jon

Reputation: 89

When one node is very slow in Hadoop Cluster?

I've hadoop cluster of 5 nodes.

I've two concerns

1) What can be done when one of the node is running or processing data very slow (Not stopped) comapre to other nodes .. ?

2) I've set up log4j to capture logs, but How can I keep logs of all nodes at Name node or at one main server .. ?

Please suggest ...!

Thanks

Upvotes: 1

Views: 498

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

To question one, it's not clear which service is slow... Datanode? Namenode? Maybe you need to increase the heap sizes of these processes, or the Dataset you've stored is heavily skewed onto that server.

You would need to install monitoring software to capture IO, CPU, network, etc metrics to really diagnose any hardware bottlenecks. From there, make sure that that one server is running latest OS patches, has latest drivers, and a similar hardware profile of other machines you're comparing against. Maybe the hard drive or NIC is failing, but without hardware diagnostic software, it'd be hard to know

For question 2, you'd again need additional software, such as Elasticsearch, to centrally collect and index your logs from many systems

Upvotes: 1

Related Questions