CODEWITHSUNDEEP
javahadoop
Kshitiz Sharma
Kshitiz Sharma

Reputation: 18627

MalformedURLException on reading file from HDFS

I have following test program to read a file from HDFS. 

public class FileReader {
    public static final String NAMENODE_IP = "172.32.17.209";
    public static final String FILE_PATH = "/notice.html";

    public static void main(String[] args) throws MalformedURLException,
            IOException {
        String url = "hdfs://" + NAMENODE_IP + FILE_PATH;

        InputStream is = new URL(url).openStream();
        InputStreamReader isr = new InputStreamReader(is);
        BufferedReader br = new BufferedReader(isr);
        String line = br.readLine();
        while(line != null) {
            System.out.println(line);
            line = br.readLine();
        }
    }
}

It is giving java.net.MalformedURLException

Exception in thread "main" java.net.MalformedURLException: unknown protocol: hdfs
    at java.net.URL.<init>(URL.java:592)
    at java.net.URL.<init>(URL.java:482)
    at java.net.URL.<init>(URL.java:431)
    at in.ksharma.hdfs.FileReader.main(FileReader.java:29)

Upvotes: 5

Views: 3505

Answers (3)

Atais
Atais

Reputation: 11285

In our case we had to combine it with other answer:
https://stackoverflow.com/a/21118824/1549135

So firstly in our HDFS setup class (Scala code):

val hadoopConfig: Configuration = new Configuration()
hadoopConfig.set("fs.hdfs.impl", classOf[DistributedFileSystem].getName)
hadoopConfig.set("fs.file.impl", classOf[LocalFileSystem].getName)

And later, like in accepted answer:
https://stackoverflow.com/a/25971334/1549135

URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory)
Try(new URL(path))

Side note:

We already had: "org.apache.hadoop" % "hadoop-hdfs" % "2.8.0" in our dependencies and it did not help.

Upvotes: 2

Jason
Jason

Reputation: 173

I get the same issue while writing a Java application for reading from hdfs on hadoop 2.6. My solution is : Add 

 hadoop-2.X/share/hadoop/hdfs/hadoop-hdfs-2.X.jar to your classpath.

Upvotes: 3

Kshitiz Sharma
Kshitiz Sharma

Reputation: 18627

Register Hadoop's Url handler. Standard Url handler won't know how to handle hdfs:// scheme.

Try this:

public static void main(String[] args) throws MalformedURLException,
            IOException {
        URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());

        String url = "hdfs://" + NAMENODE_IP + FILE_PATH;

        InputStream is = new URL(url).openStream();
        InputStreamReader isr = new InputStreamReader(is);
        BufferedReader br = new BufferedReader(isr);
        String line = br.readLine();
        while(line != null) {
            System.out.println(line);
            line = br.readLine();
        }
    }

Upvotes: 9

Related Questions