CODEWITHSUNDEEP
hadoophdfs
Rose Perrone
Rose Perrone

Reputation: 63546

Running Hadoop locally, accessing HDFS in VM. Can't find local files

Why can't HDFS read files from my local machine? The path in this exception seems to indicate that it's reading from my laptop (not the VM) but it can't find the file. The ls shows it does exist, though (see the bottom line). Please let me know what more information I can provide!

java.io.FileNotFoundException: File file:/Users/rose/bigdata/4/data/data.txt does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:468)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:380)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:139)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:335)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:746)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:83)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:478)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:671)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.secur
14/02/03 16:05:27 WARN mapred.JobClient: Error reading task outputConnection refused
14/02/03 16:05:27 WARN mapred.JobClient: Error reading task outputConnection refused
ç∂ç^C~/bigdata/4/data ls
c1.txt c2.txt data.txt vocab.txt

Upvotes: 1

Views: 1277

Answers (1)

blspeiser
blspeiser

Reputation: 91

I believe that unless you have specially configured Hadoop to ignore HDFS (core-site.xml: property fs.defaultFS set to file:///), then it will read from HDFS.

Either upload your files to HDFS (by using hdfs dfs -put <file> <destination-on-hdfs> to an actively running HDFS node), or change your Hadoop configuration. You may need to configure more than just the fs.defaultFS property to get it properly working.

I hope that helps. :)

Upvotes: 4

Related Questions