CODEWITHSUNDEEP
scalahadoopapache-sparkhdfs
Mohan Guruswamy
Mohan Guruswamy

Reputation: 21

Hdfs file list in scala

i am trying to find the list of file in hdfs directory but the code its expecting file as the input when i try to run the below code.

val TestPath2="hdfs://localhost:8020/user/hdfs/QERESULTS1.csv"
val hdfs: org.apache.hadoop.fs.FileSystem = org.apache.hadoop.fs.FileSystem.get(sc.hadoopConfiguration)
      val hadoopPath = new org.apache.hadoop.fs.Path(TestPath1)
      val recursive = true
     // val ri = hdfs.listFiles(hadoopPath, recursive)()
      //println(hdfs.getChildFileSystems)
      //hdfs.get(sc

val ri=hdfs.listFiles(hadoopPath, true)
println(ri)

Upvotes: 2

Views: 3642

Answers (1)

unsulliedcow
unsulliedcow

Reputation: 31

You should set your default filesystem to hdfs:// first, I seems like your default filesystem is file://

val conf = sc.hadoopConfiguration 
conf.set("fs.defaultFS", "hdfs://some-path")

val hdfs: org.apache.hadoop.fs.FileSystem = org.apache.hadoop.fs.FileSystem.get(conf)

...

Upvotes: 2

Related Questions