CODEWITHSUNDEEP
bashsed
labyrinth
labyrinth

Reputation: 13956

Converting DOS ^M char to Unix format failing

I'm removing DOS ^M characters from some files. If I run this sed line on a single file, it removes the characters as expected:

$ sed -i 's/^M//g' somefile.txt

However, if I run the same line inside a for loop pulling from a list of files, it does not work:

$ for i in `cat list`; do sed -i 's/^M//g' $i; done

Note that I can apply other commands instead of sed and they perform as expected on each file in sequence. Also, no errors are thrown (not even a code since echo $? returns 0). I've tried quoting $i as well.

As per the suggestion in the comments, I tried dos2unix by doing:

$ for i in `cat list`; do dos2unix $i; done

but nothing changes. Using dos2unix directly on a single file works fine. If my list files was bad, I would expect there to be at least an error code thrown...

Upvotes: 1

Views: 119

Answers (2)

labyrinth
labyrinth

Reputation: 13956

The answer was that I was I wasn't being careful enough with entering the ^M character when running the commands. As other posts on SO explain, it is necessary to type ctrl-v, ctrl-m to enter ^M as a single control character.

My issue was that I assumed that if I entered it that way initially, it was fine to highlight and paste it after that, but this isn't true for the terminal emulators I'm using (guake, tmux).

Upvotes: 0

hek2mgl
hek2mgl

Reputation: 158110

First, I don't think that sed understands ^M for carriage return. It should be \r instead.

Further, because filenames in UNIX can contain the default field delimiter of bash I suggest to use a while-read loop instead:

while read -r file ; do
    sed -i 's/\r//' "$file"
done < list

The above loop reads line-by-line. However, since files can also contain the new line character itself, mostly it is better to use find to generate the list dynamically and use it's -exec option:

find PATH_TO_FILES -type f -name 'NAME' ... -exec sed -i 's/\r//' {} \;

Btw, the g option makes no sense in the sed command since the \r will only appear once per line. That's why I've omitted it.

Upvotes: 2

Related Questions