How to remove a newline from a string in Bash

Question

I have the following variable.

echo "|$COMMAND|"

which returns

|
REBOOT|

How can I remove that first newline?

Answer 1

In Bash the 'substring replacement' feature will allow you to remove newline characters

Note that this feature is an ANSI C bashism. It may not be available on other shells and is non-posix compliant.

In Bash it is a concise alternative to the slower tr command (see below for performance comparison).

It takes the general form:

${parameter/pattern/string}

For more details see the section: Substring Replacement on tldp.org

For example:

COMMAND=$'\nREBOOT\r   \n'
echo "|${COMMAND}|"
|
   OOT
|

echo "|${COMMAND//[$'\t\r\n']}|"
|REBOOT   |

echo "|${COMMAND//[$'\t\r\n ']}|"
|REBOOT|

See Parameter Expansion and ANSI C standard in bash's man page:

man bash | sed '/^$/{ x;\|parameter/pattern/string|{s/^\n*//;p};s/.*//;h;};H;d' 

     ${parameter/pattern/string}
     ${parameter//pattern/string}
     ${parameter/#pattern/string}
     ${parameter/%pattern/string}
            Pattern  substitution.  The pattern is expanded to produce a
            pattern just as in pathname  expansion.   Parameter  is  ex‐
            panded and the longest match of pattern against its value is
            replaced with string.  string undergoes tilde expansion, pa‐
            rameter  and  variable expansion, arithmetic expansion, com‐
            mand and process substitution, and quote removal.  The match
            is  performed using the rules described under Pattern Match‐
            ing below.  In the first form above, only the first match is
            replaced.  If there are two slashes separating parameter and
            pattern (the second form above), all matches of pattern  are
            replaced  with  string.   If  pattern  is preceded by # (the
            third form above), it must match at the beginning of the ex‐
            panded value of parameter.  If pattern is preceded by % (the
            fourth form above), it must match at the end of the expanded
            value  of  parameter.   If  the expansion of string is null,
            matches of pattern are deleted.  If string is null,  matches
            of  pattern  are  deleted and the / following pattern may be
            omitted.

man bash | sed '/^$/{ x;/ANSI \+C \+standard/{s/^\n*//;p};s/.*//;h;};H;d'
man bash | sed '/^$/{ x;/ANSI[[:space:]]\+C[[:space:]]\+standard/{s/^\n*//;p};s/.*//;h;};H;d'

     Character  sequences of the form $'string' are treated as a special
     variant of single quotes.  The sequence  expands  to  string,  with
     backslash-escaped characters in string replaced as specified by the
     ANSI C standard.  Backslash escape sequences, if present,  are  de‐
     coded as follows:
            \a     alert (bell)
            \b     backspace
            \e
            \E     an escape character
            \f     form feed
            \n     new line
            \r     carriage return
            \t     horizontal tab
            \v     vertical tab
            \\     backslash
            \'     single quote
            \"     double quote
            \?     question mark
            \nnn   the  eight-bit  character  whose  value  is the octal
                   value nnn (one to three octal digits)
            \xHH   the eight-bit character whose value is the  hexadeci‐
                   mal value HH (one or two hex digits)
            \uHHHH the  Unicode (ISO/IEC 10646) character whose value is
                   the hexadecimal value HHHH (one to four hex digits)
            \UHHHHHHHH
                   the Unicode (ISO/IEC 10646) character whose value  is
                   the hexadecimal value HHHHHHHH (one to eight hex dig‐
                   its)
            \cx    a control-x character

Further...

As asked by @AlexJordan, this will suppress all specified characters. So what if $COMMAND do contain spaces...

COMMAND=$'         \n        RE BOOT      \r           \n'
echo "|$COMMAND|"
|
           BOOT      
|

read -r COMMAND <<<"${COMMAND//[$'\t\r\n']}"
echo "|$COMMAND|"
|RE BOOT|

Explanation

Answering Vulwsztyn's question:

Why does this work when the pattern is empty?

In ${COMMAND//[$'\t\r\n ']} :

• The 1st slashe / mean: Pattern substitution (following ${parameter/pattern/string} syntax)
• The pattern is /[$'\r\n '], begin with / then all matches of pattern are replaced with string
• Then, the replacement string is empty (as there a no second / followed by any string...).

1 step more further

If you try to replace nothing by something, for sample two consecutive spaces (then you could add two more spaces after replaced string in order to balance output):

echo "|${COMMAND//*()/  }  |"
|  R  E     B  O  O  T  |

Avoid fork to tr for single string!

Let compare:

COMMAND=$'\nREBOOT\r   \n'
echo ${COMMAND@Q}
$'\nREBOOT\r \n'

COMMAND=$(echo $COMMAND|tr -d '\n\t\r ')
echo ${COMMAND@Q}
'REBOOT'

Then

time for i in {1..1000};do
    COMMAND=$'\nREBOOT\r   \n'
    COMMAND=$(echo $COMMAND|tr -d '\n\t\r ')
done;echo ${COMMAND@Q}

real    0m2.785s
user    0m2.296s
sys     0m0.774s
'REBOOT'

With

COMMAND=$'\nREBOOT\r   \n'
COMMAND="${COMMAND//[$'\t\r\n ']}"
echo ${COMMAND@Q}

time for i in {1..1000};do
    COMMAND=$'\nREBOOT\r   \n'
    COMMAND="${COMMAND//[$'\t\r\n ']}"
done;echo ${COMMAND@Q}

real    0m0.006s
user    0m0.001s
sys     0m0.004s
'REBOOT'

Doing 1'000 forks to tr take more than 2700ms on my host, while same job is done in 6ms ( 464.2x faster!! ) by using built-in bash Parameter Expansion!!

Note: In fact: var=$(echo | tr x y) implie two forks, not only one! By using following syntax, you will avoid 1 (x1000) fork, so this could be a little quicker:

time for i in {1..1000};do
    COMMAND=$'\nREBOOT\r   \n'
    COMMAND=$( tr -d '\n\t\r ' <<<"$COMMAND" )
done;echo ${COMMAND@Q}

real    0m2.181s
user    0m1.590s
sys     0m0.566s
'REBOOT'

But still a lot overkill compared to the pure bash method.

Answer 2

Note that the shell already does that for you if you pass $COMMAND as a parameter instead of a string.

So you can try this:

COMMAND="a
   b
   c"

echo $COMMAND

To add the pipes, you can just use the quotes and no spaces like so:

echo "|"$COMMAND"|"

works just fine and it's a tad simpler than the other solutions. It also works with just /bin/sh.

Answer 3

Use this bashism if you don't want to spawn processes like (tr, sed or awk) for such a simple task. Bash can do that alone:

COMMAND=${COMMAND//$'\n'/}

From the documentation:

${FOO//from/to} Replace all
${FOO/from/to}  Replace first match

Answer 4

Clean your variable by removing all the linefeeds:

COMMAND=$(echo $COMMAND|tr -d '\n')

Answer 5

You can simply use echo -n "|$COMMAND|".

$ man echo

-n do not output the trailing newline

Answer 6

To address one possible root of the actual issue, there is a chance you are sourcing a crlf file.

CRLF Example:

.env (crlf)

VARIABLE_A="abc"
VARIABLE_B="def"

run.sh

#!/bin/bash
source .env
echo "$VARIABLE_A"
echo "$VARIABLE_B"
echo "$VARIABLE_A $VARIABLE_B"

Returns:

abc
def
 def

If however you convert to LF:

.env (lf)

VARIABLE_A="abc"
VARIABLE_B="def"

run.sh

#!/bin/bash
source .env
echo "$VARIABLE_A"
echo "$VARIABLE_B"
echo "$VARIABLE_A $VARIABLE_B"

Returns:

abc
def
abc def

Answer 7

Adding answer to show example of stripping multiple characters including \r using tr and using sed. And illustrating using hexdump.

In my case I had found that a command ending with awk print of the last item |awk '{print $2}' in the line included a carriage-return \r as well as quotes.

I used sed 's/["\n\r]//g' to strip both the carriage-return and quotes.

I could also have used tr -d '"\r\n'.

Interesting to note sed -z is needed if one wishes to remove \n line-feed chars.

$ COMMAND=$'\n"REBOOT"\r   \n'

$ echo "$COMMAND" |hexdump -C
00000000  0a 22 52 45 42 4f 4f 54  22 0d 20 20 20 0a 0a     |."REBOOT".   ..|

$ echo "$COMMAND" |tr -d '"\r\n' |hexdump -C
00000000  52 45 42 4f 4f 54 20 20  20                       |REBOOT   |

$ echo "$COMMAND" |sed 's/["\n\r]//g' |hexdump -C
00000000  0a 52 45 42 4f 4f 54 20  20 20 0a 0a              |.REBOOT   ..|

$ echo "$COMMAND" |sed -z 's/["\n\r]//g' |hexdump -C
00000000  52 45 42 4f 4f 54 20 20  20                       |REBOOT   |

And this is relevant: What are carriage return, linefeed, and form feed?

• CR == \r == 0x0d
• LF == \n == 0x0a

Answer 8

What worked for me was echo $testVar | tr "\n" " "

Where testVar contained my variable/script-output

Answer 9

Using bash:

echo "|${COMMAND/$'\n'}|"

^{(Note that the control character in this question is a 'newline' (\n), not a carriage return (\r); the latter would have output REBOOT| on a single line.)}

Explanation

Uses the Bash Shell Parameter Expansion ${parameter/pattern/string}:

The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. [...] If string is null, matches of pattern are deleted and the / following pattern may be omitted.

Also uses the $'' ANSI-C quoting construct to specify a newline as $'\n'. Using a newline directly would work as well, though less pretty:

echo "|${COMMAND/
}|"

Full example

#!/bin/bash
COMMAND="$'\n'REBOOT"
echo "|${COMMAND/$'\n'}|"
# Outputs |REBOOT|

Or, using newlines:

#!/bin/bash
COMMAND="
REBOOT"
echo "|${COMMAND/
}|"
# Outputs |REBOOT|

Answer 10

If you are using bash with the extglob option enabled, you can remove just the trailing whitespace via:

shopt -s extglob
COMMAND=$'\nRE BOOT\r   \n'
echo "|${COMMAND%%*([$'\t\r\n '])}|"

This outputs:

|
RE BOOT|

Or replace %% with ## to replace just the leading whitespace.

Answer 11

echo "|$COMMAND|"|tr '\n' ' '

will replace the newline (in POSIX/Unix it's not a carriage return) with a space.

To be honest I would think about switching away from bash to something more sane though. Or avoiding generating this malformed data in the first place.

Hmmm, this seems like it could be a horrible security hole as well, depending on where the data is coming from.