CODEWITHSUNDEEP
javaeclipseapachemavenhadoop
priyaranjan
priyaranjan

Reputation: 379

Not able to fetch data from HDFS using URL protocol handler

I have a problem with running the following piece of program in java(I am a beginner in java).Here the program uses the HDFS sepecific URLstreamhandlerfactory to use the appropriate protocol handler to access the HDFS. In eclipse , it does not show any error. I have put hadoop-common-2.2.0.jar in build path.

package org.hdfs.prog;

//cc URLCat Displays files from a Hadoop filesystem on standard output using a //URLStreamHandler



import java.io.InputStream;

import java.net.URL;

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;


public class URLCat {

    static {
        URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
    }

    public static void main(String[] args) throws Exception {
        InputStream in = null;
        try {
            in = new URL(args[0]).openStream();
            IOUtils.copyBytes(in, System.out, 4096, false);
        } finally {
            IOUtils.closeStream(in);
        }
    }
}

But when I am running it, I am getting class not found error, as below.

Well this clearly says its not finding one class "org.apache.commons.logging.LogFactory" during run time . To overcome this I downloaded the jar file containing the package "org.apache.commons.logging". Then again I ran the code again another class not found error.

Is there any solution which will tell me in advance what are the run time dependency jar files I will require ? Please help me out.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
    at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:165)
    at org.apache.hadoop.fs.FsUrlStreamHandlerFactory.<init>(FsUrlStreamHandlerFactory.java:54)
    at org.hdfs.prog.URLCat.<clinit>(URLCat.java:14)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
    ... 3 more
Could not find the main class: org.hdfs.prog.URLCat. Program will exit.

Upvotes: 0

Views: 1031

Answers (1)

andrershov
andrershov

Reputation: 468

I suggest you to use build tool like Maven. When you specify hadoop artifact in your pom.xml, all of its direct and indirect dependencies would be downloaded. Hadoop have a lot of dependencies and downloading all jars one by one would be time consuming.

All you need is to add maven dependency to your pom.xml, all other dependencies would be downloaded automatically:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>0.20.2</version>
</dependency>

Upvotes: 2

Related Questions