CODEWITHSUNDEEP
javahadoophdfs
MMakati
MMakati

Reputation: 703

Get the state of the hdfs namenode using java

I am creating a java application that will read the files in the directory. The problem now is how to get the state/status of the namenode since I am getting an error message:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1932)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1313)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3861)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1076)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

Please see code below:

Configuration conf = new Configuration();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/mapred-site.xml"));

System.out.println("Connecting to active name node: " + aNode);
fs = FileSystem.get(new URI(aNode),conf); 
System.out.println("Successfully connected to active name node: " + aNode);

I want to make my code more dynamic, the java will check what is the active active name node.

Upvotes: 1

Views: 2490

Answers (1)

Enes Korukcu
Enes Korukcu

Reputation: 132

I've encountered the same problem. We have a hadoop cluster with 2 hdfs nodes. One of them is active and the other one is at standby mode.

They're constantly switching roles and i have to stay connected to them using java.

I've solved it like this:

In /etc/hadoop/conf/core-site.xml there's a fs.defaultFS property. You have to create new URI from fs.defaultFS value and pass it to FileSystem.get function as parameter.

/etc/hadoop/conf/core-site.xml:

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://yourclustername-ns</value>
</property>

Code:

public static FileSystem hdfs;
try {
    Configuration conf = new Configuration();
    conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
    conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
    hdfs = FileSystem.get(new URI("hdfs://yourcluster-ns"), conf, "your-hdfs-user");
}

Usually everyone passes the address of one of the hdfs nodes to the FileSystem.get function. But nodes can change roles, they can go offline/online. So you need to pass the cluster ns address.

Upvotes: 3

Related Questions