sed command with -i option failing on Mac, but works on Linux

Question

I've successfully used the following sed command to search/replace text in Linux:

sed -i 's/old_link/new_link/g' *

However, when I try it on my Mac OS X, I get:

"command c expects \ followed by text"

I thought my Mac runs a normal BASH shell. What's up?

EDIT:

According to @High Performance, this is due to Mac sed being of a different (BSD) flavor, so my question would therefore be how do I replicate this command in BSD sed?

EDIT:

Here is an actual example that causes this:

sed -i 's/hello/gbye/g' *

Answer 1

Portable solution below

Why you get the error

The -i option (alternatively, --in-place) means that you want files edited in-place, rather than streaming the change to a new place.

Modifying a file in-place suggests a need for a backup file - and so a user-specified extension is expected after -i, but the parsing of the extension argument is handled differently under GNU sed & Mac (BSD) sed:

• GNU : "If no extension is supplied, the original file is overwritten without making a backup." - effectively, you can omit specify a file extension altogether. The extension must be supplied immediately after the -i, with no intervening space.
• Mac (BSD) : "If a zero-length extension is given, no backup will be saved." - you must supply an extension, but it can be the empty string '' if you want, to disable the backup.

So GNU & Mac will interpret this differently:

sed -i 's/hello/bye/g' just_a_file.txt

• GNU : No extension is supplied immediately after the -i, so create no backup, use s/hello/bye/g as the text-editing command, and act on the file just_a_file.txt in-place.
• Mac (BSD) : Use s/hello/bye/g as the backup file extension (!) and just_a_file.txt as the editing command (regular expression).
  Result: the command code used is j (not a valid command code for substitution, e.g. s), hence we get the error invalid command code j.

# This still create a `my_file.txt-e` backup on macOS Sonoma (14.5)
# and a `my_file.txt''` on Linux
sed -i'' -e 's/hello/bye/g' my_file.txt

Placing the extension immediately after the -i (eg -i'' or -i'.bak', without a space) is what GNU sed expects, but macOS expect a space after -i (eg -i '' or -i '.bak').

and is now accepted by Mac (BSD) sed too, though it wasn't tolerated by earlier versions (eg with Mac OS X v10.6, a space was required after -i, eg -i '.bak').

The -e parameter allows us to be explicit about where we're declaring the edit command.

Until Mac OS was updated in 2013, there wasn't

Still there isn't any portable command across GNU and Mac (BSD), as these variants all failed (with an error or unexpected backup files):

• sed -i -e ... - works on Linux but does not work on macOS as it creates -e backups
• sed -i '' -e ... - works on macOS but fails on Linux
• sed -i='' -e ... - create = backups files on macOS and Linux
• sed -i'' -e ... - create -e backups files on macOS

Portable solution

You have few options to achieve the same result on Linux and macOS, e.g.:

1. Use Perl: perl -i -pe's/old_link/new_link/g' *.

2. Use gnu-sed on macOS (Install using Homebrew)

# Install 'gnu-sed' on macOS using Homebrew
brew install gnu-sed
# Use 'gsed' instead of 'sed' on macOS.
gsed -i'' -e 's/hello/bye/g' my_file.txt

Note: On macOS, you could add the bin path of gnu-sed containing the sed command to the PATH environment variable in your shell configuration file (.zshrc).
It is best not to do this, since there may be scripts that rely on the macOS built-in version.

You can add an alias for gsed as sed using alias sed=gsed (replacing macOS sed with GNU sed) in your ~/.zshrc. This should allow you to use sed "linux-stile" in your shell and will have no effects on scripts unless they contain shopt -s expand_aliases.

If you are using sed in a script, you can try to automate switching to gsed:

#!/usr/bin/env bash
set -Eeuo pipefail

if [[ "$OSTYPE" == "darwin"* ]]; then
  # Require gnu-sed.
  if ! [ -x "$(command -v gsed)" ]; then
    echo "Error: 'gsed' is not istalled." >&2
    echo "If you are using Homebrew, install with 'brew install gnu-sed'." >&2
    exit 1
  fi
  SED_CMD=gsed
else
  SED_CMD=sed
fi

# Use '${SED_CMD}' instead of 'sed'
${SED_CMD} -i'' -e 's/hello/bye/g' my_file.txt

You can temporarily set PATH to use "gnu-sed" sed for a script:

# run a linux script containing sed without changing it
PATH="$(brew --prefix)/opt/gnu-sed/libexec/gnubin:$PATH" ./linux_script_using_sed.sh

If you are copy/pasting linux scripts, you can alias gsed to sed in the current shell:

alias sed=gsed
sed -i 's/hello/bye/g' just_a_file.txt

3. Use -i '' on macOS and BSD or -i (GNU sed) otherwise

#!/usr/bin/env bash
set -Eeuo pipefail

case "$OSTYPE" in
  darwin*|bsd*)
    echo "Using BSD sed style"
    sed_no_backup=( -i '' )
    ;; 
  *)
    echo "Using GNU sed style"
    sed_no_backup=( -i )
    ;;
esac

sed ${sed_no_backup[@]} -e 's/hello/bye/g' my_file.txt

Answer 2

An alternative portable method is to use Vim in Ex mode ex:

ex -sc '%s/old/new/g' -c 'x' file

1. -s silent mode
2. -c run command

We pass two commands:

1. -c '%s/old/new/g'

   • % select all lines
   • s replace
   • /old/new/
   • g replace all occurrences

2. -c 'x'

   • x save and exit

Be mindful that simply passing |x like this '%s/old/new/g|x' causes the editor to hang if the string is not present in the file:

Error detected while processing command line:
E486: Pattern not found: old

Using an additional -c 'x' solves that problem.

Most people want this in a find command, so here's the magical incantation:

find . \
  -type f \
  -exec ex -sc '%s/old/new/g' -c 'x' {} \;

P.S. don't mess up your git folder like I did:

find . \
  -path ./.git -prune -o \
  -type f \
  -exec ex -sc '%s/old/new/g' -c 'x' {} \;

Answer 3

I found a more compatible version of portable sed command that work for both Linux and Mac. For certain case such as using in package.json scripts section, the -i'' somehow stops working in Mac even in version Sonoma, which throws error message like "extra characters at the end of d command".

The trick is using -i='' rather than -i '' or -i'' or -i. Here is a simple example:

echo 'ABCD1234' > /tmp/test.txt
sed -i='' -e 's/ABCD/DCBA/g' /tmp/test.txt
cat /tmp/test.txt

Both Linux and Mac output the same DCBA1234.

Answer 4

Insead of calling sed with sed, I do ./bin/sed

And this is the wrapper script in my ~/project/bin/sed

#!/bin/bash

if [[ "$OSTYPE" == "darwin"* ]]; then
  exec "gsed" "$@"
else
  exec "sed" "$@"
fi

Don't forget to chmod 755 the wrapper script.

Answer 5

sed -ie 's/old_link/new_link/g' *

Works on both BSD & Linux with gnu sed

Answer 6

Here is an option in bash scripts:

#!/bin/bash

GO_OS=${GO_OS:-"linux"}

function detect_os {
    # Detect the OS name
    case "$(uname -s)" in
      Darwin)
        host_os=darwin
        ;;
      Linux)
        host_os=linux
        ;;
      *)
        echo "Unsupported host OS. Must be Linux or Mac OS X." >&2
        exit 1
        ;;
    esac

   GO_OS="${host_os}"
}

detect_os

if [ "${GO_OS}" == "darwin" ]; then
    sed -i '' -e ...
else
    sed -i -e ...
fi

Answer 7

I've created a function to handle sed difference between MacOS (tested on MacOS 10.12) and other OS:

OS=`uname`
# $(replace_in_file pattern file)
function replace_in_file() {
    if [ "$OS" = 'Darwin' ]; then
        # for MacOS
        sed -i '' -e "$1" "$2"
    else
        # for Linux and Windows
        sed -i'' -e "$1" "$2"
    fi
}

Usage:

$(replace_in_file 's,MASTER_HOST.*,MASTER_HOST='"$MASTER_IP"',' "./mysql/.env")

Where:

, is a delimeter

's,MASTER_HOST.*,MASTER_HOST='"$MASTER_IP"',' is pattern

"./mysql/.env" is path to file

Answer 8

Here's how to apply environment variables to template file (no backup need).

1. Create template with {{FOO}} for later replace.

echo "Hello {{FOO}}" > foo.conf.tmpl

2. Replace {{FOO}} with FOO variable and output to new foo.conf file

FOO="world" && sed -e "s/{{FOO}}/$FOO/g" foo.conf.tmpl > foo.conf

Working both macOS 10.12.4 and Ubuntu 14.04.5

Answer 9

As the other answers indicate, there is not a way to use sed portably across OS X and Linux without making backup files. So, I instead used this Ruby one-liner to do so:

ruby -pi -e "sub(/ $/, '')" ./config/locales/*.yml

In my case, I needed to call it from a rake task (i.e., inside a Ruby script), so I used this additional level of quoting:

sh %q{ruby -pi -e "sub(/ $/, '')" ./config/locales/*.yml}

Answer 10

Had the same problem in Mac and solved it with brew:

brew install gnu-sed

and use as

gsed SED_COMMAND

you can set as well set sed as alias to gsed (if you want):

alias sed=gsed

Answer 11

This works with both GNU and BSD versions of sed:

sed -i'' -e 's/old_link/new_link/g' *

or with backup:

sed -i'.bak' -e 's/old_link/new_link/g' *

Note missing space after -i option! (Necessary for GNU sed)

Answer 12

Sinetris' answer is right, but I use this with find command to be more specific about what files I want to change. In general this should work (tested on osx /bin/bash):

find . -name "*.smth" -exec sed -i '' 's/text1/text2/g' {} \;

In general when using sed without find in complex projects is less efficient.

Answer 13

Or, you can install the GNU version of sed in your Mac, called gsed, and use it using the standard Linux syntax.

For that, install gsed using ports (if you don't have it, get it at http://www.macports.org/) by running sudo port install gsed. Then, you can run sed -i 's/old_link/new_link/g' *

Answer 14

I believe on OS X when you use -i an extension for the backup files is required. Try:

sed -i .bak 's/hello/gbye/g' *

Using GNU sed the extension is optional.

Answer 15

Your Mac does indeed run a BASH shell, but this is more a question of which implementation of sed you are dealing with. On a Mac sed comes from BSD and is subtly different from the sed you might find on a typical Linux box. I suggest you man sed.