CODEWITHSUNDEEP
javahadoophdfs
Jcoder
Jcoder

Reputation: 30

Is there any existing API to read and write to HDFS from JAVA

Is there any existing API that exists to read/write from HDFS, along with best practices of how to implement it .

Upvotes: 1

Views: 972

Answers (1)

yoga
yoga

Reputation: 1959

Below is the code snippet

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;



String Dest = "/user/pkumar/test.xml";

 Configuration conf = new Configuration();
       FileSystem fs = FileSystem.get(URI.create(Dest),conf);
        Path path = new Path(Dest);

    if(!fs.exists(path)){
        OutputStream out = fs.create(path, new Progressable(){
            public void progress(){
                System.out.print(".");
            }
        });
        System.out.println();
        IOUtils.copyBytes(connect, out, 4096, true);
    }

use the below dependencies in your pom.xml

<dependencies>
<dependency>
     <groupId>org.apache.hadoop</groupId>
     <artifactId>hadoop-common</artifactId>
     <version>2.3.0-cdh5.1.3</version>
</dependency>
<dependency>
     <groupId>org.apache.hadoop</groupId>
     <artifactId>hadoop-client</artifactId>
     <version>2.3.0-cdh5.1.3</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.3.2</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.3.2</version>
</dependency>

</dependencies>

Upvotes: 3

Related Questions