CODEWITHSUNDEEP
hadoopmapreducedistributed-cache
Code Ninja
Code Ninja

Reputation: 23

Hadoop Distributed Cache: file not found exception

I am trying to implement K-means on MapReduce. I have uploaded initial centroid file to distributed cache

In driver class

DistributedCache.addCacheFile(new URI("GlobalCentroidFile"),conf);

In my mapper class

Path[] localFiles = DistributedCache.getLocalCacheFiles(job);
File file = new File(localFiles[0].getName());
System.out.println(" File read is "+localFiles[0].getName());
BufferedReader bufferedReader = new BufferedReader(new FileReader(file));       
System.out.println("Goin in while loop");
    ....
   // some code omitted
    ....
    } catch (IOException e) {
        System.out.println("\n"+e);
    }

output in $HADOOP_HOME/logs/ is

File read is localhostGlobalCentroidFile
java.io.FileNotFoundException: localhostGlobalCentroidFile (No such file or directory)

when I do

ganesh@ganesh-PC:~/Desktop$ hadoop fs -ls

Warning: $HADOOP_HOME is deprecated.


Found 4 items

-rw-r--r--   1 ganesh supergroup         26 2013-04-02 16:15 /user/ganesh
/GlobalCentroidFile

-rw-r--r--   1 ganesh supergroup         18 2013-04-02 16:16 /user/ganesh
/GlobalCentroidFile1

-rw-r--r--   1 ganesh supergroup        672 2013-04-02 16:15 /user/ganesh/input

drwxr-xr-x   - ganesh supergroup          0 2013-04-02 16:16 /user/ganesh/output

ganesh@ganesh-PC:~/Desktop$ hadoop fs -cat GlobalCentroidFile

Warning: $HADOOP_HOME is deprecated.


2.3    4.3

34.4    33.3

45.5    34

What might be the problem?

Upvotes: 2

Views: 2470

Answers (1)

shazin
shazin

Reputation: 21903

Following should be your code. You are trying to read a file which is available in HDFS using normal java file read constructs which will not work.

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
int bytesRead = 0;
byte[] buffer = new byte[2048];
Path inFile = new Path(argv[0]);
if (fs.exists(inFile)) {
    FSDataInputStream in = fs.open(inFile);
    while ((bytesRead = in.read(buffer)) > 0) {
        // Read Logic
    }
    in.close();
}

Upvotes: 1

Related Questions