CODEWITHSUNDEEP
jsonbashquoting
nad87563
nad87563

Reputation: 4072

Build a JSON string with Bash variables

I need to read these bash variables into my JSON string and I am not familiar with bash. any help is appreciated. 

#!/bin/sh

BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar

JSON_STRING='{"bucketname":"$BUCKET_NAME"","objectname":"$OBJECT_NAME","targetlocation":"$TARGET_LOCATION"}'


echo $JSON_STRING

Upvotes: 192

Views: 248335

Answers (21)

Adam Genshaft
Adam Genshaft

Reputation: 824

The answers here are good when the object properties are known up front. If it's a list of properties of an n length however, I find this utility a better approach:

build_json() {
  local args=("$@")
  local arg_length=${#args[@]}
  local obj="{}"

  for ((i = 0; i < "$arg_length"; i++)); do
    if [[ ${args[$i]} == --* ]] && [[ $((i + 1)) -lt $arg_length ]]; then
      local key="${args[$i]#--}"
      local value="${args[$((i + 1))]}"
      obj=$(echo "$obj" | jq "(.\"${key}\"|=\"$value\")")
    fi
  done

  echo "$obj"
}

An example use will be:

$ build_json --asdf 1 --fdsa 2
{
  "asdf": "1",
  "fdsa": "2"
}

# or

$ build_json --prop1 val1 --prop2 val2 --prop3 val3 --prop4 val4
{
  "prop1": "val1",
  "prop2": "val2",
  "prop3": "val3",
  "prop4": "val4"
}

Upvotes: 0

Vinicius Fortuna
Vinicius Fortuna

Reputation: 572

If you don't want to use jq or other external binary, you can use this escape_json_string function in Bash:

#!/bin/bash

function escape_json_string() {
  local input=$1
  for ((i = 0; i < ${#input}; i++)); do
    local char="${input:i:1}"
    local escaped="${char}"
    case "${char}" in
      $'"' ) escaped="\\\"";;
      $'\\') escaped="\\\\";;
      *)
        if [[ "${char}" < $'\x20' ]]; then
          case "${char}" in 
            $'\b') escaped="\\b";;
            $'\f') escaped="\\f";;
            $'\n') escaped="\\n";;
            $'\r') escaped="\\r";;
            $'\t') escaped="\\t";;
            *) escaped=$(printf "\u%04X" "'${char}")
          esac
        fi;;
    esac
    echo -n "${escaped}"
  done
}

BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION='/opt/My "Test"/testworkflow-2.0.1.jar'

JSON_STRING="{
\"bucketname\":\"$(escape_json_string "${BUCKET_NAME}")\",
\"objectname\":\"$(escape_json_string "${OBJECT_NAME}")\",
\"targetlocation\":\"$(escape_json_string "${TARGET_LOCATION}")\"
}"

echo "${JSON_STRING}"

Output:

{
"bucketname":"testbucket",
"objectname":"testworkflow-2.0.1.jar",
"targetlocation":"/opt/My \"Test\"/testworkflow-2.0.1.jar"
}

Upvotes: 0

James Z.M. Gao
James Z.M. Gao

Reputation: 646

with jq "@sh" we can translate jq string array to bash array:

#!/bin/bash

set -eufo pipefail
#set -x

# $1 should calculate an json array from input
jq_array() {
  jq "$1"
}

# $1 is an array name
# $2 should calculate an json array from input
read_jq_array() {
  eval unset "$1"
  eval "$(jq -r '(["'"$1"'=("] + ['"$2"' | @sh ] + [")"]) | join(" ")')"
}

dump_array() {
  printf "[%s]\n" "$@"
}

input_string='["a'\''  b", "2\"\nnew line", "1\t2\t3"]'

# view the input
jq_array '.' <<< "$input_string"

arr=()

# transform to bash array
read_jq_array arr . <<< "$input_string"

# view the result
dump_array "${arr[@]}"

The bash function read_jq_array parse a json array and eval assign to the target bash array. All special characters line new line, tab, space, quotas are kept raw as expected.

Upvotes: 0

Artem Klevtsov
Artem Klevtsov

Reputation: 9423

You can do it with jo:

#/bin/sh

BUCKET_NAME="testbucket"
OBJECT_NAME="testworkflow-2.0.1.jar"
TARGET_LOCATION="/opt/test/testworkflow-2.0.1.jar"

jo bucketname="${BUCKET_NAME}" objectname="${OBJECT_NAME}" targetlocation="${TARGET_LOCATION}"
# {"bucketname":"testbucket","objectname":"testworkflow-2.0.1.jar","targetlocation":"/opt/test/testworkflow-2.0.1.jar"}

Upvotes: 0

nm-happy-to-code
nm-happy-to-code

Reputation: 27

You can simply make a call like this to print the JSON.

#!/bin/sh
BUCKET_NAME=testbucket

OBJECT_NAME=testworkflow-2.0.1.jar

TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar

echo '{ "bucketName": "'"$BUCKET_NAME"'", "objectName": "'"$OBJECT_NAME"'", "targetLocation": "'"$TARGET_LOCATION"'" }'

or

JSON_STRING='{ "bucketName": "'"$BUCKET_NAME"'", "objectName": "'"$OBJECT_NAME"'", "targetLocation": "'"$TARGET_LOCATION"'" }'
echo $JOSN_STRING

Upvotes: 0

Guillermo L&#243;pez Alejos
Guillermo L&#243;pez Alejos

Reputation: 371

If you need to build a JSON representation where members mapped to undefined or empty variables should be ommited, then jo can help.

#!/bin/bash

BUCKET_NAME=testbucket
OBJECT_NAME=""

JO_OPTS=()

if [[ ! "${BUCKET_NAME}x" = "x" ]] ; then
        JO_OPTS+=("bucketname=${BUCKET_NAME}")
fi

if [[ ! "${OBJECT_NAME}x" = "x" ]] ; then
        JO_OPTS+=("objectname=${OBJECT_NAME}")
fi

if [[ ! "${TARGET_LOCATION}x" = "x" ]] ; then
        JO_OPTS+=("targetlocation=${TARGET_LOCATION}")
fi

jo "${JO_OPTS[@]}"

The output of the commands above would be just (note the absence of objectname and targetlocation members):

{"bucketname":"testbucket"}

Upvotes: 9

Cyrus
Cyrus

Reputation: 88563

A possibility:

#!/bin/bash 

BUCKET_NAME="testbucket"
OBJECT_NAME="testworkflow-2.0.1.jar"
TARGET_LOCATION="/opt/test/testworkflow-2.0.1.jar

# one line
JSON_STRING='{"bucketname":"'"$BUCKET_NAME"'","objectname":"'"$OBJECT_NAME"'","targetlocation":"'"$TARGET_LOCATION"'"}'

# multi-line
JSON_STRING="{
\"bucketname\":\"${BUCKET_NAME}\",
\"objectname\":\"${OBJECT_NAME}\",
\"targetlocation\":\"${TARGET_LOCATION}\"
}"

# [optional] validate the string is valid json
echo "${JSON_STRING}" | jq

Upvotes: 66

jakub.g
jakub.g

Reputation: 41218

For a general case of building JSON from bash with arbitrary inputs, many of the previous responses (even the high voted ones with jq) omit cases when the variables contain " double quote, or \n newline escape string, and you need complex string concatenation of the inputs.

When using jq you need to printf %b the input first to get the \n converted to real newlines, so that once you pass through jq you get \n back and not \\n.

I found this with version with nodejs to be quite easy to reason about if you know javascript/nodejs well:

TITLE='Title'
AUTHOR='Bob'
JSON=$( TITLE="$TITLE" AUTHOR="$AUTHOR" node -p 'JSON.stringify( {"message": `Title: ${process.env.TITLE}\n\nAuthor: ${process.env.AUTHOR}`} )' )

It's a bit verbose due to process.env. but allows to properly pass the variables from shell, and then format things inside (nodejs) backticks in a safe way.

This outputs:

printf "%s\n" "$JSON"
{"message":"Title: Title\n\nAuthor: Bob"}

(Note: when having a variable with \n always use printf "%s\n" "$VAR" and not echo "$VAR", whose output is platform-dependent! See here for details)

Similar thing with jq would be

TITLE='Title'
AUTHOR='Bob' 
MESSAGE="Title: ${TITLE}\n\nAuthor: ${AUTHOR}"
MESSAGE_ESCAPED_FOR_JQ=$(printf %b "${MESSAGE}")
JSON=$( jq '{"message": $jq_msg}' --arg jq_msg "$MESSAGE_ESCAPED_FOR_JQ" --null-input --compact-output --raw-output --monochrome-output )

(the last two params are not necessary when running in a subshell, but I just added them so that the output is then same when you run the jq command in a top-level shell).

Upvotes: 2

ADV-IT
ADV-IT

Reputation: 831

Used this for AWS Macie configuration:

JSON_CONFIG=$( jq -n \
   --arg bucket_name "$BUCKET_NAME" \
   --arg kms_key_arn "$KMS_KEY_ARN" \
   '{"s3Destination":{"bucketName":$bucket_name,"kmsKeyArn":$kms_key_arn}}'
)

aws macie2 put-classification-export-configuration --configuration "$JSON_CONFIG"

Upvotes: 0

Evy Bongers
Evy Bongers

Reputation: 641

In addition to chepner's answer, it's also possible to construct the object completely from args with this simple recipe:

BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar

JSON_STRING=$(jq -n \
                  --arg bucketname "$BUCKET_NAME" \
                  --arg objectname "$OBJECT_NAME" \
                  --arg targetlocation "$TARGET_LOCATION" \
                   '$ARGS.named')

Explanation:

  • --null-input | -n disabled reading input. From the man page: Don't read any input at all! Instead, the filter is run once using null as the input. This is useful when using jq as a simple calculator or to construct JSON data from scratch.
  • --arg name value passes values to the program as predefined variables: value is available as $name. All named arguments are also available as $ARGS.named

Because the format of $ARGS.named is already an object, jq can output it as is.

Upvotes: 54

gholk
gholk

Reputation: 91

if you have node.js and get minimist installed in global:

jc() {
    node -p "JSON.stringify(require('minimist')(process.argv), (k,v) => k=='_'?undefined:v)" -- "$@"
}
jc --key1 foo --number 12 --boolean \
    --under_score 'abc def' --'white space' '   '
# {"key1":"foo","number":12,"boolean":true,"under_score":"abc def","white space":"   "}

you can post it with curl or what:

curl --data "$(jc --type message --value 'hello world!')" \
    --header 'content-type: application/json' \
    http://server.ip/api/endpoint

be careful that minimist will parse dot:

jc --m.room.member @gholk:ccns.io
# {"m":{"room":{"member":"@gholk:ccns.io"}}}

Upvotes: 0

mahee96
mahee96

Reputation: 842

I had to work out all possible ways to deal json strings in a command request, Please look at the following code to see why using single quotes can fail if used incorrectly.

# Create Release and Tag commit in Github repository

# returns string with in-place substituted variables 

json=$(cat <<-END
    {
        "tag_name": "${version}", 
        "target_commitish": "${branch}", 
        "name": "${title}", 
        "body": "${notes}", 
        "draft": ${is_draft}, 
        "prerelease": ${is_prerelease} 
    }
END
)

# returns raw string without any substitutions
# single or double quoted delimiter - check HEREDOC specs

json=$(cat <<-!"END"   # or 'END' 
    {
        "tag_name": "${version}", 
        "target_commitish": "${branch}", 
        "name": "${title}", 
        "body": "${notes}", 
        "draft": ${is_draft}, 
        "prerelease": ${is_prerelease} 
    }
END
)

# prints fully formatted string with substituted variables as follows:

echo "${json}"  
{ 
    "tag_name" : "My_tag", 
    "target_commitish":"My_branch"
    ....
}

Note 1: Use of single vs double quotes

# enclosing in single quotes means no variable substitution 
# (treats everything as raw char literals)

echo '${json}'   
${json} 

echo '"${json}"'   
"${json}" 

# enclosing in single quotes and outer double quotes causes
# variable expansion surrounded by single quotes(treated as raw char literals).

echo "'${json}'" 
'{ 
    "tag_name" : "My_tag", 
    "target_commitish":"My_branch"
    ....
}'

Note 2: Caution with Line terminators

  • Note the json string is formatted with line terminators such as LF \n
  • or carriage return \r(if its encoded on windows it contains CRLF \r\n)
  • using (translate) tr utility from shell we can remove the line terminators if any

# following code serializes json and removes any line terminators 
# in substituted value/object variables too

json=$(echo "$json" | tr -d '\n' | tr -d '\r' )

# string enclosed in single quotes are still raw literals

echo '${json}'   
${json} 

echo '"${json}"'   
"${json}" 

# After CRLF/LF are removed

echo "'${json}'" 
'{ "tag_name" : "My_tag", "target_commitish":"My_branch" .... }'

Note 3: Formatting

  • while manipulating json string with variables, we can use combination of ' and " such as following, if we want to protect some raw literals using outer double quotes to have in place substirution/string interpolation:
# mixing ' and " 

username=admin
password=pass

echo "$username:$password"
admin:pass

echo "$username"':'"$password"
admin:pass

echo "$username"'[${delimiter}]'"$password"
admin[${delimiter}]pass

Note 4: Using in a command

  • Following curl request already removes existing \n (ie serializes json)
response=$(curl -i \
            --user ${username}:${api_token} \
            -X POST \
            -H 'Accept: application/vnd.github.v3+json' \
            -d "$json" \
            "https://api.github.com/repos/${username}/${repository}/releases" \
            --output /dev/null \
            --write-out "%{http_code}" \
            --silent
          )

So when using it for command variables, validate if it is properly formatted before using it :)

Upvotes: 28

tamerlaha
tamerlaha

Reputation: 1990

You could use envsubst:

  export VAR="some_value_here"
  echo '{"test":"$VAR"}' | envsubst > json.json

also it might be a "template" file:

//json.template
{"var": "$VALUE", "another_var":"$ANOTHER_VALUE"}

So after you could do:

export VALUE="some_value_here"
export ANOTHER_VALUE="something_else"
cat  json.template | envsubst > misha.json

Upvotes: 2

Steve Elliott
Steve Elliott

Reputation: 19

These solutions come a little late but I think they are inherently simpler that previous suggestions (avoiding the complications of quoting and escaping).

    BUCKET_NAME=testbucket
    OBJECT_NAME=testworkflow-2.0.1.jar
    TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar
    
    # Initial unsuccessful solution
    JSON_STRING='{"bucketname":"$BUCKET_NAME","objectname":"$OBJECT_NAME","targetlocation":"$TARGET_LOCATION"}'
    echo $JSON_STRING 
    
    # If your substitution variables have NO whitespace this is sufficient
    JSON_STRING=$(tr -d [:space:] <<JSON
    {"bucketname":"$BUCKET_NAME","objectname":"$OBJECT_NAME","targetlocation":"$TARGET_LOCATION"}
    JSON
    )
    echo $JSON_STRING 
    
    # If your substitution variables are more general and maybe have whitespace this works
    JSON_STRING=$(jq -c . <<JSON
    {"bucketname":"$BUCKET_NAME","objectname":"$OBJECT_NAME","targetlocation":"$TARGET_LOCATION"}
    JSON
    )
    echo $JSON_STRING 
    
    #... A change in layout could also make it more maintainable
    JSON_STRING=$(jq -c . <<JSON
    {
       "bucketname" : "$BUCKET_NAME",
       "objectname" : "$OBJECT_NAME",
       "targetlocation" : "$TARGET_LOCATION"
    }
    JSON
    )
    echo $JSON_STRING

Upvotes: 1

kiko283
kiko283

Reputation: 530

Bash will not insert variables into a single-quote string. In order to get the variables bash needs a double-quote string. You need to use double-quote string for the JSON and just escape double-quote characters inside JSON string. Example:

#!/bin/sh

BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar

JSON_STRING="{\"bucketname\":\"$BUCKET_NAME\",\"objectname\":\"$OBJECT_NAME\",\"targetlocation\":\"$TARGET_LOCATION\"}"


echo $JSON_STRING

Upvotes: 0

binaryfunt
binaryfunt

Reputation: 7127

To build upon Hao's answer using NodeJS: you can split up the lines, and use the -p option which saves having to use console.log.

JSON_STRING=$(node -pe "
  JSON.stringify({
    bucketname: process.env.BUCKET_NAME,
    objectname: process.env.OBJECT_NAME,
    targetlocation: process.env.TARGET_LOCATION
  });
")

An inconvenience is that you need to export the variables beforehand, i.e.

export BUCKET_NAME=testbucket
# etc.

Note: You might be thinking, why use process.env? Why not just use single quotes and have bucketname: '$BUCKET_NAME', etc so bash inserts the variables? The reason is that using process.env is safer - if you don't have control over the contents of $TARGET_LOCATION it could inject JavaScript into your node command and do malicious things (by closing the single quote, e.g. the $TARGET_LOCATION string contents could be '}); /* Here I can run commands to delete files! */; console.log({'a': 'b. On the other hand, process.env takes care of sanitising the input.

Upvotes: 1

Hao
Hao

Reputation: 51

For Node.js Developer, or if you have node environment installed, you can try this:

JSON_STRING=$(node -e "console.log(JSON.stringify({bucketname: $BUCKET_NAME, objectname: $OBJECT_NAME, targetlocation: $TARGET_LOCATION}))")

Advantage of this method is you can easily convert very complicated JSON Object (like object contains array, or if you need int value instead of string) to JSON String without worrying about invalid json error.

Disadvantage is it's relying on Node.js environment.

Upvotes: 2

Prasad Wargad
Prasad Wargad

Reputation: 765

can be done following way:

JSON_STRING='{"bucketname":"'$BUCKET_NAME'","objectname":"'$OBJECT_NAME'","targetlocation":"'$TARGET_LOCATION'"}'

Upvotes: 5

chepner
chepner

Reputation: 530920

You are better off using a program like jq to generate the JSON, if you don't know ahead of time if the contents of the variables are properly escaped for inclusion in JSON. Otherwise, you will just end up with invalid JSON for your trouble.

BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar

JSON_STRING=$( jq -n \
                  --arg bn "$BUCKET_NAME" \
                  --arg on "$OBJECT_NAME" \
                  --arg tl "$TARGET_LOCATION" \
                  '{bucketname: $bn, objectname: $on, targetlocation: $tl}' )

Upvotes: 339

Diego Torres Milano
Diego Torres Milano

Reputation: 69188

You can use printf:

JSON_FMT='{"bucketname":"%s","objectname":"%s","targetlocation":"%s"}\n'
printf "$JSON_FMT" "$BUCKET_NAME" "$OBJECT_NAME" "$TARGET_LOCATION"

much clear and simpler

Upvotes: 83

glenn jackman
glenn jackman

Reputation: 246744

First, don't use ALL_CAPS_VARNAMES: it's too easy to accidentally overwrite a crucial shell variable (like PATH)

Mixing single and double quotes in shell strings can be a hassle. In this case, I'd use printf:

bucket_name=testbucket
object_name=testworkflow-2.0.1.jar
target_location=/opt/test/testworkflow-2.0.1.jar
template='{"bucketname":"%s","objectname":"%s","targetlocation":"%s"}'

json_string=$(printf "$template" "$BUCKET_NAME" "$OBJECT_NAME" "$TARGET_LOCATION")

echo "$json_string"

For homework, read this page carefully: Security implications of forgetting to quote a variable in bash/POSIX shells

A note on creating JSON with string concatenation: there are edge cases. For example, if any of your strings contain double quotes, you can broken JSON:

$ bucket_name='a "string with quotes"'
$ printf '{"bucket":"%s"}\n' "$bucket_name"
{"bucket":"a "string with quotes""}

Do do this more safely with bash, we need to escape that string's double quotes:

$ printf '{"bucket":"%s"}\n' "${bucket_name//\"/\\\"}"
{"bucket":"a \"string with quotes\""}

Upvotes: 20

Related Questions