CODEWITHSUNDEEP
scalaapache-spark
Madhavi msn
Madhavi msn

Reputation: 21

How to move all files in a directory to another in hadoop

I am trying to move all files from a directory to another directory within HDFS using spark scala. In short i need to find way to use hadoop fs -mv command in a programmatic way. As i am new to hadoop it would be great if someone could help.

Upvotes: 1

Views: 2277

Answers (1)

mgaido
mgaido

Reputation: 3055

In order to perform operation on HDFS, you just need to get a FileSystem instance, via

val fs = FileSystem.get(sparkContext.hadoopConfiguration)

then you can use its methods to do what you want on HDFS. If you want to move a file, look at the rename method.

Upvotes: 1

Related Questions