How to redirect and append both standard output and standard error to a file with Bash

Question

To redirect standard output to a truncated file in Bash, I know to use:

cmd > file.txt

To redirect standard output in Bash, appending to a file, I know to use:

cmd >> file.txt

To redirect both standard output and standard error to a truncated file, I know to use:

cmd &> file.txt

How do I redirect both standard output and standard error appending to a file? cmd &>> file.txt did not work for me.

Answer 1

Some remarks and useful tricks

Introduction

From What does " 2>&1 " mean?, I will use in this answer the command:

ls -ld /tmp /tnt

for populating simultaneously both STDIN and STDERR. (In the hope you don't have any entry in your root, called tnt.)

Redirections from script himself

You could plan redirections from the script itself:

#!/bin/bash

exec 1>>logfile.txt
exec 2>&1

/bin/ls -ld /tmp /tnt

Running this will create/append logfile.txt, containing:

/bin/ls: cannot access '/tnt': No such file or directory
drwxrwxrwt 2 root root 4096 Apr  5 11:20 /tmp

Or

#!/bin/bash

exec 1>>logfile.txt
exec 2>>errfile.txt

/bin/ls -ld /tmp /tnt

While create or append standard output to logfile.txt and create or append errors output to errfile.txt.

Log to many different files

You could create two different logfiles, appending to one overall log and recreating another last log:

#!/bin/bash

if [ -e lastlog.txt ] ;then
    mv -f lastlog.txt lastlog.old
fi
exec 1> >(tee -a overall.log /dev/tty >lastlog.txt)
exec 2>&1

ls -ld /tnt /tmp

Running this script will

• if lastlog.txt already exist, rename them to lastlog.old (overwriting lastlog.old if they exist).
• create a new lastlog.txt.
• append everything to overall.log
• output everything to the terminal.

Simple and combined logs

#!/bin/bash

[ -e lastlog.txt ] && mv -f lastlog.txt lastlog.old
[ -e lasterr.txt ] && mv -f lasterr.txt lasterr.old

exec 1> >(tee -a overall.log combined.log /dev/tty >lastlog.txt)
exec 2> >(tee -a overall.err combined.log /dev/tty >lasterr.txt)

ls -ld /tnt /tmp

So you have

• lastlog.txt last run log file
• lasterr.txt last run error file
• lastlog.old previous run log file
• lasterr.old previous run error file
• overall.log appended overall log file
• overall.err appended overall error file
• combined.log appended overall error and log combined file.
• still output to the terminal

Same, using timestamped log filenames:

#!/bin/bash

sTime=${EPOCHSECONDS}
for pre in log err; do
   printf -v ${pre}file '%s-%(%Y%m%d%H%M%S)T-%08x.txt' "$pre" "$sTime" $$
done

exec 1> >(tee -a overall.log combined.log /dev/tty >"$logfile")
exec 2> >(tee -a overall.err combined.log /dev/tty >"$errfile")

ls -ld /t{nt,mp}

• I use $sTime in order to ensure both file will present exactly same timestamp. (If $EPOCHSECONDS is invoked two times, they could differ between each expansion!)
• I add formed 8 characters hexadecimal representation of currend pid: $$ in order to ensure unicity.

after 3rd run, you must found 9 files:

ls -ltr
-rw-r--r--  1 user user   49 19 nov 10:36 log-20231119103611-00120649.txt
-rw-r--r--  1 user user   73 19 nov 10:36 err-20231119103611-00120649.txt
-rw-r--r--  1 user user   73 19 nov 10:36 err-20231119103634-001207b8.txt
-rw-r--r--  1 user user   49 19 nov 10:36 log-20231119103634-001207b8.txt
-rw-r--r--  1 user user  147 19 nov 10:40 overall.log
-rw-r--r--  1 user user  219 19 nov 10:40 overall.err
-rw-r--r--  1 user user   49 19 nov 10:40 log-20231119104000-001216f0.txt
-rw-r--r--  1 user user   73 19 nov 10:40 err-20231119104000-001216f0.txt
-rw-r--r--  1 user user  366 19 nov 10:40 combined.log

And for interactive session, use stdbuf:

Regarding Fonic' comment and after some test, I have to agree: with tee, stdbuf is useless. But ...

If you plan to use this in *interactive* shell, you must tell `tee` to not buffering his input/output:

# Source this to multi-log your session
[ -e lasterr.txt ] && mv -f lasterr.txt lasterr.old
[ -e lastlog.txt ] && mv -f lastlog.txt lastlog.old
exec 2> >(exec stdbuf -i0 -o0 tee -a overall.err combined.log /dev/tty >lasterr.txt)
exec 1> >(exec stdbuf -i0 -o0 tee -a overall.log combined.log /dev/tty >lastlog.txt)

Once sourced this, you could try:

ls -ld /tnt /tmp

More complex sample

From my 3 remarks about how to Convert Unix timestamp to a date string

I've used more complex command to parse and reassemble squid's log in real time: As each line begin by an UNIX EPOCH with milliseconds, I split the line on 1st dot, add @ symbol before EPOCH SECONDS to pass them to date -f - +%F\ %T then reassemble date's output and the rest of line with a dot by using paste -d ..

exec {datesfd}<> <(:)
tail -f /var/log/squid/access.log |
    tee >(
        exec sed -u 's/^\([0-9]\+\)\..*/@\1/'|
            stdbuf -o0 date -f - +%F\ %T >&$datesfd
    ) |
        sed -u 's/^[0-9]\+\.//' |
        paste -d . /dev/fd/$datesfd -

With date, stdbuf was required...

Some explanations about exec and stdbuf commands:

• Running forks by using $(...) or <(...) is done by running subshell wich will execute binaries in another subshell (subsubshell). The exec command tell shell that there are not further command in script to be run, so binary (stdbuf ... tee) will be executed as replacement process, at same level (no need to reserve more memory for running another sub-process).

  From bash's man page (man -P'less +/^\ *exec\ ' bash):

      exec [-cl] [-a name] [command [arguments]]
             If  command  is  specified,  it  replaces the
             shell.  No new process is created....
  

  This is not really needed, but reduce system footprint.

• From stdbuf's man page:

  NAME
         stdbuf  -  Run COMMAND, with modified buffering
         operations for its standard streams.
  

  This will tell system to use unbuffered I/O for tee command. So all outputs will be updated immediately, when some input are coming.

Answer 2

This is terribly good!

Redirect the output to log file and stdout within the current script.

Refer to https://stackoverflow.com/a/314678/5449346, very simple and clean, it redirects all the script's output to the log file and stdout, including the scripts called in the script:

exec > >(tee -a "logs/logdata.log") 2>&1 prints the logs on the screen as well as writes them into a file – shriyog Feb 2, 2017 at 9:20

Typically we would place one of these at or near the top of the script. Scripts that parse their command lines would do the redirection after parsing.

Send stdout to a file

exec > file with stderr

exec > file
exec 2>&1 append both stdout and stderr to file

exec >> file exec 2>&1 As Jonathan Leffler mentioned in his comment:

exec has two separate jobs. The first one is to replace the currently executing shell (script) with a new program. The other is changing the I/O redirections in the current shell. This is distinguished by having no argument to exec.

Answer 3

There are two ways to do this, depending on your Bash version.

The classic and portable (Bash pre-4) way is:

cmd >> outfile 2>&1

A nonportable way, starting with Bash 4 is

cmd &>> outfile

(analog to &> outfile)

For good coding style, you should

• decide if portability is a concern (then use the classic way)
• decide if portability even to Bash pre-4 is a concern (then use the classic way)
• no matter which syntax you use, don't change it within the same script (confusion!)

If your script already starts with #!/bin/sh (no matter if intended or not), then the Bash 4 solution, and in general any Bash-specific code, is not the way to go.

Also remember that Bash 4 &>> is just shorter syntax — it does not introduce any new functionality or anything like that.

The syntax is (beside other redirection syntax) described in the Bash hackers wiki.

Answer 4

Another approach:

If using older versions of Bash where &>> isn't available, you also can do:

(cmd 2>&1) >> file.txt

This spawns a subshell, so it's less efficient than the traditional approach of cmd >> file.txt 2>&1, and it consequently won't work for commands that need to modify the current shell (e.g. cd, pushd), but this approach feels more natural and understandable to me:

1. Redirect standard error to standard output.
2. Redirect the new standard output by appending to a file.

Also, the parentheses remove any ambiguity of order, especially if you want to pipe standard output and standard error to another command instead.

To avoid starting a subshell, you instead could use curly braces instead of parentheses to create a group command:

{ cmd 2>&1; } >> file.txt

(Note that a semicolon (or newline) is required to terminate the group command.)

Answer 5

This should work fine:

your_command 2>&1 | tee -a file.txt

It will store all logs in file.txt as well as dump them in the terminal.

Answer 6

Try this:

You_command 1> output.log  2>&1

Your usage of &> x.file does work in Bash 4. Sorry for that: (

Here comes some additional tips.

0, 1, 2, ..., 9 are file descriptors in bash.

0 stands for standard input, 1 stands for standard output, 2 stands for standard error. 3~9 is spare for any other temporary usage.

Any file descriptor can be redirected to other file descriptor or file by using operator > or >>(append).

Usage: <file_descriptor> > <filename | &file_descriptor>

Please see the reference in Chapter 20. I/O Redirection.

Answer 7

In Bash 4 (as well as Z shell (zsh) 4.3.11):

cmd &>> outfile

just out of box.

Answer 8

cmd >>file.txt 2>&1

Bash executes the redirects from left to right as follows:

1. >>file.txt: Open file.txt in append mode and redirect stdout there.
2. 2>&1: Redirect stderr to "where stdout is currently going". In this case, that is a file opened in append mode. In other words, the &1 reuses the file descriptor which stdout currently uses.

Answer 9

In Bash you can also explicitly specify your redirects to different files:

cmd >log.out 2>log_error.out

Appending would be:

cmd >>log.out 2>>log_error.out