CODEWITHSUNDEEP
hadoophdfs
haochi zhang
haochi zhang

Reputation: 113

How to move or copy file in HDFS by using JAVA API

I want to copy file in THE SAME HDFS ,just like copy file from HDFS://abc:9000/user/a.txt to HDFS://abc:9000/user/123/

Can I do that by using JAVA API? Thanks

Upvotes: 5

Views: 19301

Answers (2)

akshay jain
akshay jain

Reputation: 669

If you want to move files from directory it is little bit tricky below code done same task for me !!
val conf = new org.apache.hadoop.conf.Configuration()
    val src:Path = new org.apache.hadoop.fs.Path(hdfsDirectory)
    val fs = FileSystem.get(src.toUri,conf)
    val srcPath: Path = new Path("hdfs://sourcePath/")
    val srcFs =FileSystem.get(srcPath.toUri,conf)
    val dstPath:Path =new Path("hdfs://targetPath/")
    val dstFs =FileSystem.get(dstPath.toUri,conf)
    val exists = fs.exists(new org.apache.hadoop.fs.Path(hdfsDirectory))
    val status:Array[FileStatus] = fs.listStatus(new Path(hdfsDirectory))
    if (status.length>0) {
      status.foreach(x => {
        println("My files: " + x.getPath)
        FileUtil.copy(srcFs, x.getPath, dstFs, dstPath, true, conf)
        println("Files moved !!" +x.getPath)
      }
      )}
    else{
      println("No Files Found !!")
    }

Upvotes: 0

Serhiy
Serhiy

Reputation: 4141

FileUtil provides a method for copying files.

Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://abc:9000");
FileSystem filesystem = FileSystem.get(configuration);
FileUtil.copy(filesystem, new Path("src/path"), filesystem, new Path("dst/path"), false, configuration);

If you need to copy it to another cluster, just make a new Configuration and setup and new FileSystem.

Upvotes: 21

Related Questions