CODEWITHSUNDEEP
bashshellshdatabricksdatabricks-notebook
Sarath Subramanian
Sarath Subramanian

Reputation: 21381

Handle error message from the cat command

I am using Databricks Notebook to execute Shell script using the %sh and I want to capture error message into err variable and then print the error message if the cat command fails (for eg, purposefully fail if the header_file directory doesn't exist). In the below code, if the cat command fails, it is coming inside the if statement and printing the error code correctly, but it is not showing the error from the err variable. The error message is showing as Error msg : and not showing why cat command failed.

err=$(cat /dbfs/mnt/devl/header_file/*.csv /dbfs/mnt/devl/data_files/*.csv > /dbfs/mnt/devl/output.txt 2>&1)
RC=$?
if [ $RC -ne 0 ]; then
  echo "Error code : $RC"
  echo "Error msg : $err"
fi

I checked if the issue is with the cat command and tried to replicate using ls command. In ls command, it is displaying the error message correctly as Error msg : ls: cannot access '/dbfs/mnt/devl/header_file/*.csv': No such file or directory. Is this due to some limitation with the cat command?

err=$(ls /dbfs/mnt/devl/header_file/*.csv 2>&1)
RC=$?
if [ $RC -ne 0 ]; then
  echo "Error code : $RC"
  echo "Error msg : $err"
fi

Note : My question is not related how to handle file/folder not found. The intention of this question is to display the actual error message that says why cat command failed for any kind of failure from the cat command.

Upvotes: 1

Views: 154

Answers (3)

Mika Feiler
Mika Feiler

Reputation: 516

The redirection order is tricky. Like h0tw1r3 says here: https://unix.stackexchange.com/a/298528/96266 and links https://www.gnu.org/software/bash/manual/html_node/Redirections.html which says:

Note that the order of redirections is significant. For example, the command

ls > dirlist 2>&1

directs both standard output (file descriptor 1) and standard error (file descriptor 2) to the file dirlist, while the command

ls 2>&1 > dirlist

directs only the standard output to file dirlist, because the standard error was made a copy of the standard output before the standard output was redirected to dirlist.

Your question is the opposite of that question.

$ ls a b ga
ls: nie ma dostępu do 'a': No such file or directory
ls: nie ma dostępu do 'b': No such file or directory
ga:
$ ls a b ga 2>&1 > g | wc
      2      22     108
$ cat g
ga:

Upvotes: 1

KamilCuk
KamilCuk

Reputation: 141748

> is handled by shell, not by cat. cat is not printing any messages. Add a command block for the inner shell to have the redirection.

err=$(
    {
        cat \
            /dbfs/mnt/devl/header_file/*.csv \
            /dbfs/mnt/devl/data_files/*.csv \
            > /dbfs/mnt/devl/output.txt
    } 2>&1
)
RC=$?
if ((!RC)); then
  echo "Error code : $RC"
  echo "Error msg : $err"
fi

Upvotes: 3

Romeo Ninov
Romeo Ninov

Reputation: 7265

You can resolve the issue by changing the places of STDOUT and STDERR redirects:

err=$(cat /dbfs/mnt/devl/header_file/*.csv /dbfs/mnt/devl/data_files/*.csv 2>&1 > /dbfs/mnt/devl/output.txt)

This will send the STDERR to the original STDOUT handler (in your case this is a err variable) and redirect the original STDOUT handler to file.

Upvotes: 1

Related Questions