CODEWITHSUNDEEP
bashreplacesedcharacter-encoding
user198530
user198530

Reputation: 69

Very weird behavior using sed

I have a big problem doing a script: basically, I read a line from files. All lines are made of 3 to 8 characters contiguous (no space). Then I used sed to replace those lines inside a pattern (aka "var" in my minimal script below)

var="iao"
for m in `more meshing/junction_names.txt`
do 
    echo $m 
    echo -n $m  | xxd -ps | sed 's/[[:xdigit:]]\{2\}/\\x&/g'
    echo $var |sed "s/a/b/"
    echo $var |sed "s/a/$m/"
done

Now these are the first 3 record of my output (they are all the same anyway). I am using linux. According kate, all files are encoded UTF-8. Very weird huh? Any idea why that is is welcome.

J_LEAK 
\x4a\x5f\x4c\x45\x41\x4b\x0d
ibo
oJ_LEAK
JO_1
\x4a\x4f\x5f\x31\x0d
ibo
oJO_1
JPL2_F
\x4a\x50\x4c\x32\x5f\x46\x0d
ibo
oJPL2_F
JF_PL2

Upvotes: 1

Views: 291

Answers (1)

tripleee
tripleee

Reputation: 189317

Your input file contains DOS carriage returns (or possibly, the absurd attempt to read it with more introduces them). The hex dump shows this clearly; every value ends with \x0d which translates to a control code which causes the terminal to jump the cursor back to the beginning of the line.

This is a massive FAQ and you can find many examples of how to troubleshoot this basic problem, including in the bash tag wiki.

Tangentially, you should always quote strings unless you specifically require the shell to perform wildcard expansion and whitespace tokenization on the value; and Bash has built-ins to avoid the inelegant and somewhat error-prone echo | sed. Finally, don't read lines with for.

var="iao"
tr -d '\015' <meshing/junction_names.txt |
while read -r m; do  # don't use a for loop
    echo "$m"        # quote! 
    echo -n "$m"  | xxd -ps | sed 's/[[:xdigit:]]\{2\}/\\x&/g'
    echo "${var/a/b}" # quote; use Bash built-in substitution mechanism
    echo "${var/a/$m}"
done

Perhaps you want to remove the carriage returns once and for all, and then just use while read .... done <fixed-file instead of the tr pipeline.

Upvotes: 1

Related Questions