CODEWITHSUNDEEP
bashfilesed
dorien
dorien

Reputation: 5407

Weird ^M and no correct line breaks in csv file, trying to remove with sed

I am trying to process a bunch of files that were saved in a very weird way.

They start like this: 

started recording,2016-08-16 15:12:10.661^M
stopped recording,2016-08-16 15:16:29.373^M
item number,16^M
baseline,no^M
item,something^M
item,something^M
item,X             ^Mitem,something                ^Mitem,12,of 14  ^something,10,of 10 ^Mitem,10^Mitem,good
^M
time,item,continue^M

Text editor Kate shows the file correctly, but vim doesn't and Python doesn't read it correctly. There should be a line break after item, X and after 14.

I basically need to remove the double tabs between them and replace with a comma and line break. I did that using the code below:

#replace double tab by tab
find ./ -type f -exec sed -i 's/\t\t/\t/g' {} \;

#replace tab by comma and enter
find ./ -type f -exec sed -i 's/\t/,\n\t/g' {} \;

#remove lines that contain only whitespace
find ./ -type f -exec sed -i sed '/^\s*$/d'{} \;

But it's still weird: 

started recording,2016-08-11 15:08:24.926^M
stopped recording,2016-08-11 15:12:12.671^M
item,10^M
item,no^M
item,something^M
item,something
Condition,X,
        ^Mitem,relax,
        ^Msomething,10,of 14 ^Msomething ^Mitem,10,
        ^Mitem,good^M
time,item,continue^M

It basically needs to be a two column csv file. How can I remove the strange behaviour?

--update, after running sed -i 's/\r//g' filename it still doesn't split playlist to another line.

started recording,2016-08-16 14:56:49.520
stopped recording,2016-08-16 15:03:09.203
item,16
item,no
item,something
item,something
item,X             item,something          something,9,of 14  item,7,of 10 Titem,10item,good

time,item,continue

The problem remains with line: item,X item,something somethi...

The line breaks still did not come back.

Upvotes: 0

Views: 230

Answers (1)

heemayl
heemayl

Reputation: 42107

It is CR (Carriage Return), not tab, do:

sed -i 's/\r//g' file.txt

Or use dos2unix which is specifically designed for this:

dos2unix file.txt

Upvotes: 3

Related Questions