CODEWITHSUNDEEP
javahadoopmapreduce
Harsh
Harsh

Reputation: 265

Not able to open HDFS file from mapper Hadoop

I searched a lot but failed to find a solution to this problem. Actually the file I want to access is in HDFS, but not in input path (the path which was input to the map/reduce job). And I want to access it from mapper. The hdfs path specified in the input path is perfectly accessible from mapper but the other hdfs files are not.

INside mapper:-

FileSystem FS1=FileSystem.get(conf);
Path path=new Path(""+FS1.getHomeDirectory());
FSDataInputStream fsdis=FS1.open(path);

RESULTS IN the following ERROR: java.io.IOException : Cannot open filename /user/hadoop

Thanks in advance, Harsh

Upvotes: 1

Views: 5551

Answers (2)

tiru
tiru

Reputation: 11

can u try this once

try {
    FileSystem fs = FileSystem.get (new Configuration ());
    FileStatus[] status = fs.listStatus (new Path ("hdfs://jp.seka.com:9000/user/jeka/in"));
    for (int i=0;i < status.length;i++) {
       BufferedReader br = new BufferedReader (new InputStreamReader (fs.open (status[i].getPath())));
       String line;
       line = br.readLine();
       while (line != null) {
           System.out.println (line);
           line=br.readLine ();
       }
    }
} catch (Exception e) {
      System.out.println("File not found");
}

Upvotes: 1

Mateusz Dymczyk
Mateusz Dymczyk

Reputation: 15141

I remember using this tutorial to get something similar working. You can give it a try, it has only a few difference tho what you've written but still it might help...

@Edit: ah and I just noticed (after reading the comments) that you are trying to open FS1.getHomeDirectory() and that is a directory. You should point out to a file not a directory, I think (you can check it out in the linked tutorial under "Reading data from a file").

Upvotes: 1

Related Questions