How to delete HDFS folder having windows special characters (^M) in the name

Question

I wrote a shell script to create hdfs folders in windows 7 and ran on Linux server. Now, hdfs folders got created but with special character ^M at the end of the name(probably carriage return). It doesn't show up in Linux but i can see when the 'ls' output is redirected to a file. I should have run dos2unix before running this script. However now I am not able to delete folders with ^M. Could someone assist on how to delete these folders.

Answer 1

Just a supplementary answers fo @SachinJ.

TL;DR

$ hdfs dfs -rm -r -f $(hdfs dfs -ls /path/to/dir | sed '<LINE_NUMBER>q;d' | awk '{print $<FILE_NAME_COLUM_NUMBER>}')

should be replace to line number of file you want to delete in the output of hdfs dfs -ls /path/to/dir.

Here is the example.

Details

Suppose your hdfs dir like this

$ hdfs dfs -ls /path/to/dir
Found 5 items
drwxr-xr-x   - test supergroup          0 2019-08-22 10:41 /path/to/dir/dir1
drwxr-xr-x   - test supergroup          0 2019-07-11 15:35 /path/to/dir/dir2
drwxr-xr-x   - test supergroup          0 2019-07-05 17:53 /path/to/dir/dir3
drwxr-xr-x   - test supergroup          0 2019-08-22 11:28 /path/to/dir/dirtodelete
drwxr-xr-x   - test supergroup          0 2019-07-26 11:07 /path/to/dir/dir4

When you ls from it, the screen output looks just ok.

But you can't select it

$ hdfs dfs -ls /path/to/dir/dirtodelete
ls: `/path/to/dir/dirtodelete': No such file or directory

$ hdfs dfs -ls /path/to/dir/dirtodelete*
ls: `/path/to/dir/dirtodelete*': No such file or directory

What's more, when output ls result to file and use vim to read, it shows like following

$ hdfs dfs -ls /path/to/dir > tmp

$ vim tmp
Found 5 items
drwxr-xr-x   - test supergroup          0 2019-08-22 10:41 /path/to/dir/dir1
drwxr-xr-x   - test supergroup          0 2019-07-11 15:35 /path/to/dir/dir2
drwxr-xr-x   - test supergroup          0 2019-07-05 17:53 /path/to/dir/dir3
drwxr-xr-x   - test supergroup          0 2019-08-22 11:28 /path/to/dir/dirtodelete^M^M
drwxr-xr-x   - test supergroup          0 2019-07-26 11:07 /path/to/dir/dir4

What is "^M", it's a CARRIAGE RETURN (CR). More info here

Linux \n(LF) eq to Windows \r\n(CRLF)

This problem occurs edit same file in Windows and Linux.

So, we just to use correct filename, then we can delete it .But it can't be copy from the screen.

Here sed command works!

ls output as following

$ hdfs dfs -ls /path/to/dir
Found 5 items
drwxr-xr-x   - test supergroup          0 2019-08-22 10:41 /path/to/dir/dir1
drwxr-xr-x   - test supergroup          0 2019-07-11 15:35 /path/to/dir/dir2
drwxr-xr-x   - test supergroup          0 2019-07-05 17:53 /path/to/dir/dir3
drwxr-xr-x   - test supergroup          0 2019-08-22 11:28 /path/to/dir/dirtodelete
drwxr-xr-x   - test supergroup          0 2019-07-26 11:07 /path/to/dir/dir4

the filename is on line 5 so hdfs dfs -ls /path/to/dir | sed '5q;d' will cut the line we need. sed '5q;d' means read the first 5 line and quit, delete former lines, so it selects 5th line.

Then we can use awk the select filename column, index form 1, so column number is 8.

so just write the command

$ hdfs dfs -ls /path/to/dir/ | sed '5q;d' | awk '{print $8}'
/path/to/dir/dirtodelete

Then we can delete it.

$ hdfs dfs -rm -r -f $(hdfs dfs -ls /path/to/dir/ | sed '5q;d' | awk '{print $8}')

Answer 2

Sometimes wildchar may not work ( rm filename* ), better use the below option.

rm -r $(ls | sed '<LINE_NUMER>q;d')

Replace with line number in the output of ls command.