change json file by bash script

Question

I need your help to solve the following problem: I have a JSON file that looks like this:

{
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}

how can I add and remove a new key (i.e "key4": "value4") by bash script? I see also the issue to add or remove a comma at the end of last key in the file before adding or removing the new one.

Thank you

Answer 1

Your best bet is to use a JSON CLI such as jq:

• On Debian-based systems such as Ubuntu, you can install it via
  sudo apt-get install jq
• On macOS, with Homebrew (http://brew.sh/) installed, use
  brew install jq

Examples, based on the following input string - output is to stdout:

jsonStr='{ "key1": "value1", "key2": "value2", "key3": "value3" }'

Remove "key3":

jq 'del(.key3)' <<<"$jsonStr"

Add property "key4" with value "value4":

jq '. + { "key4": "value4" }' <<<"$jsonStr"

Change the value of existing property "key1" to "new-value1":

jq '.key1 = "new-value1"' <<<"$jsonStr"

A more robust alternative ^{thanks, Lars Kiesow} :
If you pass the new value with --arg, jq takes care of properly escaping the value:

jq '.key1 = $newVal' --arg newVal '3 " of rain' <<<"$jsonStr"

---

If you want to update a JSON file in place (conceptually speaking), using the example of deleting "key3":

# Create test file.
echo '{ "key1": "value1", "key2": "value2", "key3": "value3" }' > test.json

# Remove "key3" and write results back to test.json (recreate it with result).
jq -c 'del(.key3)' test.json > tmp.$$.json && mv tmp.$$.json test.json

You cannot replace the input file directly, so the result is written to a temporary file that replaces the input file on success.

Note the -c option, which produces compact rather than pretty-printed JSON.

For all options and commands, see the manual at https://jqlang.github.io/jq/manual/.

Answer 2

to change a file in place, use the sponge command

echo '{ "k": "old value" }' >f.json

cat f.json | jq '.k = $v' --arg v 'new value' | sponge f.json

see also: jq issue Edit files in place #105

alternative to jq: jaq

echo '{ "k": "old value" }' >f.json

jaq -i '.k = $v' --arg v 'new value' f.json

... but jaq has less features than jq

Answer 3

Here's a pure bash example, including the "comma issue".

#!/bin/bash
# This bash script just uses the sed command to 
#   replace/insert a new key at/before/after an 
#   existing key in a json file 
# The comma issue:
# - replace: with/without, as previous entry
# - before: always add
# - after: add before, if there was none
SED_CMD="/tmp/sed_cmd.tmp"
JSFILE1="./data1.json"
JSFILE2="./data2.json"
JSFILE3="./data3.json"
SEARCH_KEY="key3"
# create json input file
echo -e '{\n\t"key1": "value1",\n\t"key2": "value2",\n\t"key3": "value3"\n}' > $JSFILE1
echo -e "input:"
cat $JSFILE1
# duplicate twice
cp $JSFILE1 $JSFILE2 && cp $JSFILE1 $JSFILE3
# find the SEARCH_KEY and store the complete line to SEARCH_LINE 
SEARCH_LINE=`cat data.json | grep $SEARCH_KEY`
echo "SEARCH_LINE=>$SEARCH_LINE<"
# replace SEARCH_LINE
IS_COMMA=`echo $SEARCH_LINE | grep ","`
[ -z "$IS_COMMA" ] && \
    echo "s+$SEARCH_LINE+\t\"keyNew\": \"New\"+g" > $SED_CMD || \
    echo "s+$SEARCH_LINE+\t\"keyNew\": \"New\",+g" > $SED_CMD
sed -i -f $SED_CMD $JSFILE1
echo -e "replace:"
cat $JSFILE1
# insert before SEARCH_LINE
echo "s+$SEARCH_LINE+\t\"keyNew\": \"New\",\n$SEARCH_LINE+g" > $SED_CMD
sed -i -f $SED_CMD $JSFILE2
echo -e "before:"
cat $JSFILE2
# insert after SEARCH_LINE
IS_COMMA=`echo $SEARCH_LINE | grep ","`
[ -z "$IS_COMMA" ] && \
    echo "s+$SEARCH_LINE+$SEARCH_LINE,\n\t\"keyNew\": \"New\"+g" > $SED_CMD || \
    echo "s+$SEARCH_LINE+$SEARCH_LINE\n\t\"keyNew\": \"New\",+g" > $SED_CMD
sed -i -f $SED_CMD $JSFILE3
echo -e "after:"
cat $JSFILE3
exit 0

Answer 4

how can I add and remove a new key (i.e "key4": "value4") by bash script?

Using a dedicated JSON tool, like xidel, would be a better idea than to use pure Bash functions.

Add a new attribute-value pair

xidel -s '{"a":1,"b":2,"c":3}' -e '($json).d:=4'                 # dot notation
xidel -s '{"a":1,"b":2,"c":3}' -e '{|$json,{"d":4}|}'            # JSONiq (deprecated)
xidel -s '{"a":1,"b":2,"c":3}' -e 'map:put($json,"d",4)'         # XQuery
xidel -s '{"a":1,"b":2,"c":3}' -e 'map:merge(($json,{"d":4}))'   # XQuery
{
  "a": 1,
  "b": 2,
  "c": 3,
  "d": 4
}

Remove the attribute-value pair "c":3

xidel -s '{"a":1,"b":2,"c":3}' --xmlns:jnlib="http://jsoniq.org/function-library" -e 'jnlib:remove-keys($json,"c")'   # JSONiq (deprecated)
xidel -s '{"a":1,"b":2,"c":3}' -e 'map:remove($json,"c")'   # XQuery
{
  "a": 1,
  "b": 2
}

Change value of "c" attribute to 4

xidel -s '{"a":1,"b":2,"c":3}' -e '($json).c:=4'
xidel -s '{"a":1,"b":2,"c":3}' -e 'map:put($json,"c",4)'
xidel -s '{"a":1,"b":2,"c":3}' -e 'map:merge(($json,{"c":4}),{"duplicates":"use-last"})'
{
  "a": 1,
  "b": 2,
  "c": 4
}

Answer 5

Building off Lenny's answer, we can use node's -p option, which evaluates the given script and writes the output to stdout.

Using the spread operator for easy modification gives:

node -p "JSON.stringify({...require('./data.json'), key4: 'value4'}, null, 2)" > data.json

Answer 6

Not the answer for everyone, but if you already happen to have NodeJs installed in your system, you can use it to easily manipulate JSON.

eg:

#!/usr/bin/env bash
jsonFile=$1;

node > out_${jsonFile} <<EOF
//Read data
var data = require('./${jsonFile}');

//Manipulate data
delete data.key3
data.key4 = 'new value!';

//Output data
console.log(JSON.stringify(data));

EOF

Heck, if you only need to do JSON manipulation and you have node (ie: You don't really need any other bash functionality) you could directly write a script using node as the interpreter:

#! /usr/bin/env node
var data = require('./'+ process.argv[2]);
/*manipulate*/
console.log(JSON.stringify(data));