How to run Hadoop HDFS command from java code

Question

I'm new in Hadoop! How can I run some hdfs commands from Java code? I've been testing successfully mapreduce with java code and hdfs commands directly from cloudera vm's terminal but now I'd like to learn how to do it with java code. I've been looking for any materials where to learn but I haven't found yet. Thanks

Answer 1

@HbnKing I tried running your code but I kept getting errors. This is the error i got

 java.io.IOException: Cannot run program "your": CreateProcess error=2, The system cannot 
 find the file specified
       at java.lang.ProcessBuilder.start(Unknown Source)
       at java.lang.Runtime.exec(Unknown Source)
       at java.lang.Runtime.exec(Unknown Source)
       at java.lang.Runtime.exec(Unknowenter code heren Source)
       at jrs.main(jrs.java:5)

Answer 2

You can use FileSystem API in java code to perform Hdfs commands. https://hadoop.apache.org/docs/r2.8.2/api/org/apache/hadoop/fs/FileSystem.html

Please find the following sample code.

package com.hadoop.FilesystemClasses;
import java.io.IOException;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.log4j.Logger;
import com.hadoop.Constants.Constants;
public class HdfsFileSystemTasks {

public static Logger logger = Logger.getLogger(HdfsFileSystemTasks.class
        .getName());

public FileSystem configureFilesystem(String coreSitePath,
        String hdfsSitePath) {
    FileSystem fileSystem = null;

    try {
        Configuration conf = new Configuration();
        Path hdfsCoreSitePath = new Path(coreSitePath);
        Path hdfsHDFSSitePath = new Path(hdfsSitePath);
        conf.addResource(hdfsCoreSitePath);
        conf.addResource(hdfsHDFSSitePath);

        fileSystem = FileSystem.get(conf);
        return fileSystem;
    } catch (Exception ex) { 
        ex.printStackTrace();
        return fileSystem;
    }
}

    public String writeToHDFS(FileSystem fileSystem, String sourcePath,
        String destinationPath) {
    try {
        Path inputPath = new Path(sourcePath);
        Path outputPath = new Path(destinationPath);
        fileSystem.copyFromLocalFile(inputPath, outputPath);
        return Constants.SUCCESS;
    } catch (IOException ex) {
        ex.printStackTrace();
        return Constants.FAILURE;
    }
}

 public String readFileFromHdfs(FileSystem fileSystem, String hdfsStorePath,
        String localSystemPath) {
    try {
        Path hdfsPath = new Path(hdfsStorePath);
        Path localPath = new Path(localSystemPath);
        fileSystem.copyToLocalFile(hdfsPath, localPath);
        return Constants.SUCCESS;
    } catch (IOException ex) {
        ex.printStackTrace();
        return Constants.FAILURE;
    }
}

public String deleteHdfsDirectory(FileSystem fileSystem,
        String hdfsStorePath) {
    try {
        Path hdfsPath = new Path(hdfsStorePath);

        if (fileSystem.exists(hdfsPath)) {
            fileSystem.delete(hdfsPath);
            logger.info("Directory{} Deleted Successfully "
                    + hdfsPath);
        } else {
            logger.info("Input Directory{} does not Exists " + hdfsPath);
        }

        return Constants.SUCCESS;
    } catch (Exception ex) {
        System.out
                .println("Some exception occurred while reading file from hdfs");
        ex.printStackTrace();
        return Constants.FAILURE;
    }
}

public String deleteLocalDirectory(FileSystem fileSystem,
        String localStorePath) {
    try {
        Path localPath = new Path(localStorePath);

        if (fileSystem.exists(localPath)) {
            fileSystem.delete(localPath);
            logger.info("Input Directory{} Deleted Successfully "
                    + localPath);
        } else {
            logger.info("Input Directory{} does not Exists " + localPath);
        }
        return Constants.SUCCESS;
    } catch (Exception ex) {
        System.out
                .println("Some exception occurred while reading file from hdfs");
        ex.printStackTrace();
        return Constants.FAILURE;
    }
}


public void closeFileSystem(FileSystem fileSystem) {
    try {
        fileSystem.close();
    } catch (Exception ex) {
        ex.printStackTrace();
        System.out.println("Unable to close Hadoop filesystem : " + ex);
    }
  }
}

package com.hadoop.FileSystemTasks;

import com.hadoop.Constants.HDFSParameters;
import com.hadoop.Constants.HdfsFilesConstants;
import com.hadoop.Constants.LocalFilesConstants;
import com.hadoop.FilesystemClasses.HdfsFileSystemTasks;
import org.apache.hadoop.fs.FileSystem;
import org.apache.log4j.Logger;
public class ExecuteFileSystemTasks {

public static Logger logger = Logger.getLogger(ExecuteFileSystemTasks.class
        .getName());

public static void main(String[] args) {

    HdfsFileSystemTasks hdfsFileSystemTasks = new HdfsFileSystemTasks();
    FileSystem fileSystem = hdfsFileSystemTasks.configureFilesystem(
            HDFSParameters.CORE_SITE_XML_PATH,
            HDFSParameters.HDFS_SITE_XML_PATH);

    logger.info("File System Object {} " + fileSystem);

    String fileWriteStatus = hdfsFileSystemTasks.writeToHDFS(fileSystem,
            LocalFilesConstants.SALES_DATA_LOCAL_PATH,
            HdfsFilesConstants.HDFS_SOURCE_DATA_PATH);

    logger.info("File Write Status{} " + fileWriteStatus);

    String filereadStatus = hdfsFileSystemTasks.readFileFromHdfs(
            fileSystem, HdfsFilesConstants.HDFS_DESTINATION_DATA_PATH
                    + "/MR_Job_Res2/part-r-00000",
            LocalFilesConstants.MR_RESULTS_LOCALL_PATH);
    logger.info("File Read Status{} " + filereadStatus);

    String deleteDirStatus = hdfsFileSystemTasks.deleteHdfsDirectory(
            fileSystem, HdfsFilesConstants.HDFS_DESTINATION_DATA_PATH
                    + "/MR_Job_Res2");

    hdfsFileSystemTasks.closeFileSystem(fileSystem);

 }

}

Answer 3

As mentioned by Jagrut, you can use FileSystem API in your java code to interact with hdfs command. Below is the sample code where i am trying to check if a particular directory exists in hdfs or not. If exists, then remove that hdfs directory.

    Configuration conf = new Configuration();
    Job job = new Job(conf,"HDFS Connect");

    FileSystem fs = FileSystem.get(conf);
    Path outputPath = new Path("/user/cloudera/hdfsPath");
    if(fs.exists(outputPath))
        fs.delete(outputPath);

You can also refer to given blogs for further reference -

https://dzone.com/articles/working-with-the-hadoop-file-system-api, https://hadoop.apache.org/docs/r2.8.2/api/org/apache/hadoop/fs/FileSystem.html https://blog.knoldus.com/2017/04/16/working-with-hadoop-filesystem-api/

Answer 4

You can use the FileSystem API in your Java code to interact with HDFS.

Answer 5

I think this may be help to you

I use it execute shell command well .here is the java example

public class JavaRunShell {
    public static void main(String[] args){  
        try {  
            String shpath="  your command";
            Process ps = Runtime.getRuntime().exec(shpath);  
            ps.waitFor();  

            }  
        catch (Exception e) {  
            e.printStackTrace();  
            }  
    } 
}