CODEWITHSUNDEEP
javaeclipsehadoopcloudera-quickstart-vm
Free Man
Free Man

Reputation: 195

Cloudera Quickstart VM illegalArguementException: Wrong FS: hdfs: expected: file:

I have a simple java code to copy a text file from my local to the hdfs. I am using cloudera's quickstart virtual machine. 

Configuration conf = new Configuration();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/home/cloudera/workspace/Downloader/output/data.txt"), 
                          new Path("hdfs://quickstart.cloudera:8020/user/cloudera/"));

I get this error after running this code:

Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://quickstart.cloudera:8020/user/cloudera, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at org.apache.hadoop.fs.LocalFileSystem.pathToFile(LocalFileSystem.java:61)
at org.apache.hadoop.fs.LocalFileSystem.exists(LocalFileSystem.java:51)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:355)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:211)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:67)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1143)

What could I be doing wrong?

Upvotes: 1

Views: 700

Answers (2)

Free Man
Free Man

Reputation: 195

I resolved this problem. You have to be careful with the kind of jar files you add to your classpath especially when working with cloudera quickstart vm. If available, use jar files provided by cloudera. They can be found in this folder: /usr/lib/hadoop/client/ This code will work fine without any problems.

Configuration conf = new Configuration();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/home/cloudera/workspace/Downloader/output/data.txt"), 
                      new Path("hdfs://quickstart.cloudera:8020/user/cloudera/"));

Upvotes: 1

Fortran
Fortran

Reputation: 2336

Not correct path("hdfs://quickstart.cloudera:8020/user/cloudera/" use this example:

Configuration conf = getConf();
System.out.println("fs.default.name : - " + conf.get("fs.default.name"));
// It prints uri  as : hdfs://10.214.15.165:9000 or something...
String uri = conf.get("fs.default.name");
FileSystem fs = FileSystem.get(uri,getConf());

Upvotes: 0

Related Questions