CODEWITHSUNDEEP
linuxbashloggingsh
Dimitri Kopriwa
Dimitri Kopriwa

Reputation: 14375

How to use set -x without showing stdout?

Within CI, I am running a bash script that calls many bash scripts.

./internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}" > /dev/null

This doest not disable the stdout returned by the script.

The Gitlabi-CI runners stop logging after 100MB of log, It says Job's log exceeded limit of 10240000 bytes.

I know the log script can only grow up.

How can I optimize the output log size?

I don't need to have all the stdout, I can have stderr but then it will be a long running script without information.

Is there a way to display the commands which is running like when doing set -x?

Edit

Reading the answers, I was not able to solve my issue. I need to add that I am using nodejs to run the bash script that run the long bash script.

This is how I call my node script within .gitlab-ci.yml:

scripts:
   - node my_script.js

Within my_script.js, I have:

exports.handler = () => {
  const ls = spawn('bash', [path.join(__dirname, 'release.sh')], { stdio: 'inherit' });
  ls.on('close', (code) => {
    if (code !== 0) {
      console.log(`ps process exited with code ${code}`);
      process.exitCode = code;
    }
  });
};

Within my_script.sh, I have:

./internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}" > /dev/null

Upvotes: 1

Views: 861

Answers (2)

tripleee
tripleee

Reputation: 189477

You can selectively redirect file handles with exec.

exec >stdout 2>stderr

This however loses the connection to the terminal, so there is no simple way to output anything to the terminal after this point.

You can instead duplicate a file handle with m>&n where m is the number of the file descriptor to duplicate and n is the number of the new one (choose a big number like 99 to not accidentally clobber an existing handle).

exec 98<&1  # stdout
exec 99<&2  # stderr
exec >/dev/null 2>&1
:

To re-enable output,

exec 1<&98 2<&99

If you redirected to a temporary file instead of /dev/null you could obviously now show the tail of those files to the caller.

tail -n 100 "$TMPDIR"/stdout "$TMPDIR"/stderr

(On a shared server, probably use mktemp to create a unique temporary directory at the beginning of your script; static hard-coded file names make it impossible to run two builds at the same time.)

As you usually can't predict where the next error will happen, probably put all of this in a wrapper script which performs the redirection, runs the build, and finally displays the tail end of the temporary log files. Some build servers probably want to see some signs of life in the log file every few minutes, so perhaps tail a few lines every once in a while in a loop, too.

On the other hand, if there is just a single build command, the whole build job's stdout and stderr can simply be redirected to a log file, and you don't need to exec things back and forth. If you need to enable output selectively for portions of the script, use exec as above; but for wholesale redirection, just redirect the one command.

In summary, maybe your build script would look something like this.

#!/bin/sh

t=$(mktemp -t -d cibuild.XXXXXXXX) || exit

trap 'kill $buildpid; wait $buildpid; tail -n 500 "$t"/*; rm -rf "$t"' 0 1 2 3 5 15

# Your original commands here
${initial_process_wd}/internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}">"$t"/stdout 2>"$t"/stderr &

buildpid=$!
while kill -0 $buildpid; do
    sleep 180
    date
    tail -n 1 "$t"/*
done
wait

A flaw with this approach is that you lose timing information. A proper solution woud let you see when each line was produced, and display standard output and standard error intermixed in the order the messages were printed, perhaps with visible time stamps, and even with coloring hints (red time stamps for stderr?)

Upvotes: 2

dibery
dibery

Reputation: 3460

Option 1

If your script will output the error message to stderr, you can ignore all output to stdout by using command > /dev/null, where /dev/null is a black hole that will take away any output to it.

Option 2

If there's any pattern on your error message, you can use grep to filter out those error messages.

Edit 1:

To show the command that is running, you can supply -x command to bash; therefore, your command will be

bash -x ${initial_process_wd}/internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}" > /dev/null

bash will print the command executed to stderr

Edit 2:

If you want to reduce the size of the output file, you can pass it to gzip by using ${initial_process_wd}/internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}" | gzip > logfile.

To read the content of the logfile, you can use zcat logfile.

Upvotes: 0

Related Questions