CODEWITHSUNDEEP
scalahadoopapache-spark
Anji
Anji

Reputation: 305

Returning a Path from HDFS in Scala

I want to return the Path of the file where I am giving a folder and searching for "part" file in that folder.

def listDirectoriesGetPartFile(folderPath: String): org.apache.hadoop.fs.Path  {

val path = new Path(folderPath)
 if (fileSystem.isDirectory(path)) {
  val st = fileSystem.listStatus(path)
  for (i <- 0 until st.length) {
    if (st(i).getPath.getName.toString().trim().contains("part")) {
      st(i).getPath
    }
  }

 }
 ????
}

I want to return the part-xxx file.

How can I achieve it?

Upvotes: 0

Views: 1329

Answers (1)

Tzach Zohar
Tzach Zohar

Reputation: 37852

Here are a few options (in ascending order of compliance with Scala's idioms). I'll assume you want this to return null (although highly not recommended) in case no matching file is found:

Using a mutable var:

def listDirectoriesGetPartFile(folderPath: String): Path = {
  val path = new Path(folderPath)
  var result: Path = null // use a var with a default value
  if (fileSystem.isDirectory(path)) {
    val st = fileSystem.listStatus(path)
    // update the value of result if we found what we're looking for
    result = st.map(_.getPath).find(_.getName.trim.contains("part")).orNull
  }
  result // return the var
}

Using if-else in Scala, everything is an expression, so an if-else expression can be the returned value:

def listDirectoriesGetPartFile(folderPath: String): Path = {
  val path = new Path(folderPath)
  if (fileSystem.isDirectory(path)) {
    val st = fileSystem.listStatus(path)
    st.map(_.getPath).find(_.getName.trim.contains("part")).orNull
  } else {
    null
  }
}

Changing the signature to return an Option: since this method might find the path you're looking for, but might not (if given path isn't a folder, or if no such file exists in the folder), it's best if the return type conveys that posibility - and this is where Scala's Option can be used. For convenience, I'll also replace the if-else with pattern matching, but that's not a must:

def listDirectoriesGetPartFile(folderPath: String): Option[Path] = {
  new Path(folderPath) match {
    case p if fileSystem.isDirectory(p) => fileSystem.listStatus(p)
      .map(_.getPath)
      .find(_.getName.trim.contains("part"))
    case _ => None
  }
}

Upvotes: 1

Related Questions