CODEWITHSUNDEEP
javahadoophadoop2
Padmanabhan Vijendran
Padmanabhan Vijendran

Reputation: 306

Read a hadoop config Files from Unix Path in Java Program

I am developing an Java Application and this application is saving a result data to HDFS. The java Application should run in my windows machine.

As of now, I copied the Hadoop config files such as core-site, hdfs-site.xml in to my windows machine and testing it. Below is my configuration Code Snippet.

config.addResource(new Path("C:\\Hadoop\\core-site.xml"));
config.addResource(new Path("C:\\Hadoop\\hdfs-site.xml"));
config.addResource(new Path("C:\\Hadoop\\mapred-site.xml"));
config.addResource(new Path("C:\\Hadoop\\yarn-site.xml"));

Is there any way I can load config files directly from unix server from my Windows Machine like below.

config.addResource(new Path("/opt/hdp/2.3.4.7-4/hadoop/conf/core-site.xml"));
config.addResource(new Path("/opt/hdp/2.3.4.7-4/hadoop/conf/hdfs-site.xml"));
config.addResource(new Path("/opt/hdp/2.3.4.7-4/hadoop/conf/mapred-site.xml"));
config.addResource(new Path("/opt/hdp/2.3.4.7-4/hadoop/conf/yarn-site.xml"));

Kindly help me.

Upvotes: 1

Views: 715

Answers (2)

catweasle
catweasle

Reputation: 46

It looks like you are trying to embed a hadoop/hdfs instance. Most hdfs instances are likely to be multibox perhaps in a data centre. If you include the appropriate jars, and setup permissions you should be able to interact with hdfs via the namenode using urls:

hdfs://namenode:port/path/to/your/file

If it is for testing then spinning up a local instance independent from the prod cluster makes sense but it probably needs it's own independent config.

Upvotes: 0

Jim Garrison
Jim Garrison

Reputation: 86774

You could theoretically accomplish this with some work. There are two possibilities, SAMBA or NFS. In both cases, the Unix server would have to be running the server component and be configured to export the desired filesystem, and you would need authentication credentials that grant you access to the exported filesystem.

For SAMBA you might be able to use the suggestions in this question to map the network drive. You'll have to write code to detect if the drive is already mapped, and cope with assigning a drive letter.

For NFS you will need to find an NFS client library for Java.

I suppose a third possibility is to copy the files from the Unix server via FTP or (better) SCP.

Upvotes: 1

Related Questions