CODEWITHSUNDEEP
hadoophdfs
wsb3383
wsb3383

Reputation: 3881

HDFS: How do you list files recursively?

How do you, through Java, list all files (recursively) under a certain path in HDFS. I went through the API and noticed FileSystem.listFiles(Path,boolean) but it looks like that method doesn't exist in my instance of FileSystem when I initialize it.

Upvotes: 21

Views: 37360

Answers (4)

Pierre Louis
Pierre Louis

Reputation: 408

Using the hdfs cli with -R option

hdfs dfs -ls -R /path/to/

Upvotes: 0

Ramineni Ravi Teja
Ramineni Ravi Teja

Reputation: 3906

Use -R followed by ls command to list files/directorires recursively.

hadoop fs -ls -R Path/Of/File

Possible attributes for ls command are

-d : Directories are listed as plain files.

-h "Formats the sizes of files in a human-readable fashion rather than a number of bytes.

-R "Recursively list the contents of directories.

Upvotes: 14

Chris White
Chris White

Reputation: 30089

You can look at the source for org.apache.hadoop.fs.FsShell.ls(FileStatus, FileSystem, boolean, boolean) for your version of hadoop - this is what is called when you perform a hadoop fs -lsr path from the command line

Upvotes: 16

Lyle Z
Lyle Z

Reputation: 1363

hadoop-user@hadoop-desk ~/hadoop
$ bin/hadoop fs -lsr /user/someone_else/myfiles

-rw-r--r--   1 hadoop-user supergroup          0 2013-11-26 02:09 /user/someone_else/myfiles/file1.txt

-rw-r--r--   1 hadoop-user supergroup          0 2013-11-26 02:09 /user/someone_else/myfiles/file2.txt

drwxr-xr-x   - hadoop-user supergroup          0 2013-11-26 02:09 /user/someone_else/myfiles/subdir

-rw-r--r--   1 hadoop-user supergroup          0 2013-11-26 02:09 /user/someone_else/myfiles/subdir/anotherFile.txt

Upvotes: 0

Related Questions