CODEWITHSUNDEEP
bashshellshstdout
Harshit Kushwaha
Harshit Kushwaha

Reputation: 73

format shell script logs and write to a file

I need to log all outputs / error logs from a shell script into a file with a standard format.

For example for file script.sh:

echo "I want to be a json message!"
echo "I also want to be a json message!"

executed like below:

./script.sh >> jsonmessages.log

should give me a file content like:

{timestamp: '2020-12-31 00:00:00', message: 'I want to be a json message!'}
{timestamp: '2020-12-31 00:00:00', message: 'I also want to be a json message!'}

Upvotes: 3

Views: 2788

Answers (3)

Harshit Kushwaha
Harshit Kushwaha

Reputation: 73

Thanks for the solutions.

I was able to write logs from script executions to the log file with below implementation:

  1. Creating a json logger

      json_logger() {
         # Values for json
         current_timestamp="$(date +\"%Y-%m-%dT%H:%M:%S\")"
         param1="$param1value"
         param2="$param2value"
         log_level=$1
         script_name=$(basename $2 .sh)

         # Create json and re-direct to log file
         jq --raw-input --compact-output \
         '{
                 "@timestamp": '$current_timestamp',
                 "parameter1": "'$param1'",
                 "parameter2": "'$param2'",
                 "log-level": "'$log_level'",
                 "logger": "'$script_name'",
                 "message": .
         }' >> /var/log/app.log 2>&1 
     }

  2. And then executing scripts as below

       "script.sh" 2>&1 | json_logger "INFO" {job_name or script_name}

Upvotes: 2

Léa Gris
Léa Gris

Reputation: 19555

The best choice to reliably translate a shell string into a valid JSON string is to use a JSON parser/formatter. The most popular one is jq.

Here is an implementation of a timestamped JSON message logger using jq:

#!/usr/bin/env bash

json_logger() {
  while read -r msg || [ -n "$msg" ]; do
    jq \
      -nc \
      --arg msg "$msg" \
      '{ "timestamp": (now | strftime("%Y-%m-%d %H:%M:%S")), "message": $msg }'
  done
}


----------

EDIT:

As KamilCuk wrote:

We can do better without slow bash loop - it's just jq --raw-input '{ "timestamp": (now | strftime("%Y-%m-%d %H:%M:%S")), "message": . }'

So here is an improved json_logger:

json_logger() {
  jq \
    --raw-input \
    --compact-output \
    '{ "timestamp": (now | strftime("%Y-%m-%d %H:%M:%S")), "message": . }'
}

Addendum:

Harshit Kushwaha wrote:

For example, I need to print the method name in json from where the json_logger was called. How can I modify the json_logger and how to use it then in echo?

Here is an implementation with adding the method name if provided as an argument to the json_logger function:

#!/usr/bin/env bash

IFS= read -r -d '' __JQ_LOGGER_SCRIPT <<'JQSCRIPT'
{
  "timestamp": now | strftime("%Y-%m-%d %H:%M:%S"),
  "message": .
} |
  if ($name | length) != 0
  then
    . + { "method": $name }
  else
    .
  end
JQSCRIPT

json_logger() {
  jq \
    --raw-input \
    --compact-output \
    --arg name "$1" \
    "$__JQ_LOGGER_SCRIPT"
}

echo "I want to be a json message!" | json_logger
echo "I also want to be a json message!" | json_logger echo

printf %s $'I am a message with "quoted text" and some special characters: \'\t\7\42\\\'; that can only be properly converted to JSON with a JSON formatter and parser.' | json_logger printf

Produces this JSON outputs:

{"timestamp":"2021-01-29 14:02:46","message":"I want to be a json message!"}
{"timestamp":"2021-01-29 14:02:46","message":"I also want to be a json message!","method":"echo"}
{"timestamp":"2021-01-29 14:02:46","message":"I am a message with \"quoted text\" and some special characters: '\t\u0007\"\\'; that can only be properly converted to JSON with a JSON formatter and parser.","method":"printf"}

Upvotes: 4

Raman Sailopal
Raman Sailopal

Reputation: 12877

./script.sh | awk -v squot="'" '{ print "{\"timestamp\": \""strftime("%Y-%m-%d %H:%M:%S")"\", \"message\": \""$0"\"}" }' mess | jq >> jsonmessages.log

Using GNU awk, print the timestamp in the required format using awk's strftime function, along with double quotes and the other data in the format required. Pipe through to jq to ensure valid json

Upvotes: 1

Related Questions