Bash/JQ parse error: Expected separator between values at line 1, column 254

Question

I'm getting a parse error: while executing the function

Is this something to do with jq or bash error?

generate_readable_output() {
  mkdir -p smoke-test-logs/tmp
  counter=1
  error_length=`jq length smoke-test-logs/error-log.json`
  echo "[" > smoke-test-logs/tmp/filds-output
  jq -c '.[]' smoke-test-logs/error-log.json | while read i; do
    msg=$(echo $i | jq -r '.msg')
    type=$(echo $i | jq -r '.name')
    echo "{\"title\": \"$type\",\"value\": \"$msg\",\"short\": false}," >> smoke-test-logs/tmp/filds-output;
    if [ $counter -eq 20 ]
      then
        break
    fi ;
    counter=$(expr $counter + 1)
  done
  if [ $error_length -gt 20 ]
    then echo "{\"title\": \"There are more...\",\"value\": \"There are $error_length issues need to take necessary actions immediately.\",\"short\": false}," >> smoke-test-logs/tmp/filds-output
  fi ;
  echo $(sed '$ s/.$//' smoke-test-logs/tmp/filds-output) > smoke-test-logs/tmp/filds-output
  echo "]" >> smoke-test-logs/tmp/filds-output
}

sample error-log.json content

[{"name":"single_schedule","hostname":"ip-172-31-23-92","pid":22408,"level":50,"msg":"Exception found while processing : Property address: Unit 43, 8-14 Fullerton Street, Woollahra Nsw 2025 Property Id: 95 Lease Id: 29 ErrorCode: 400 ErrorMessage 400 - \"[{\\\"errorCode\\\":400002,\\\"message\\\":\\\"Field Validation Error\\\",\\\"details\\\":\\\"Missing mandatory field dbc.\\\",\\\"type\\\":\\\"REQUEST_ERROR\\\",\\\"field\\\":\\\"dbc\\\"}]\"","time":"2019-07-26T10:57:02.079Z","v":0},
{"name":"single_schedule","hostname":"ip-172-31-23-92","pid":22408,"level":50,"msg":"Exception found while processing : Property address: 8 Chunooma Road, North Wahroonga Nsw 2076 Property Id: 96 Lease Id: 30 ErrorCode: 400 ErrorMessage 400 - \"[{\\\"errorCode\\\":400002,\\\"message\\\":\\\"Field Validation Error\\\",\\\"details\\\":\\\"Missing mandatory field dbc.\\\",\\\"type\\\":\\\"REQUEST_ERROR\\\",\\\"field\\\":\\\"dbc\\\"}]\"","time":"2019-07-26T10:57:03.287Z","v":0},
{"name":"single_schedule","hostname":"ip-172-31-23-92","pid":22408,"level":50,"msg":"Exception found while processing : Property address: Unit 17, 92 Parraween Street, Cremorne Nsw 2090 Property Id: 111 Lease Id: 38 ErrorCode: 400 ErrorMessage 400 - \"[{\\\"errorCode\\\":400002,\\\"message\\\":\\\"Field Validation Error\\\",\\\"details\\\":\\\"Missing mandatory field dbc.\\\",\\\"type\\\":\\\"REQUEST_ERROR\\\",\\\"field\\\":\\\"dbc\\\"}]\"","time":"2019-07-26T10:57:05.402Z","v":0},
{"name":"single_schedule","hostname":"ip-172-31-23-92","pid":22408,"level":50,"msg":"Exception found while processing : Property address: Unit 72, 1-3 Delmar Parade, Dee Why Nsw 2099 Property Id: 112 Lease Id: 41 ErrorCode: 400 ErrorMessage 400 - \"[{\\\"errorCode\\\":400002,\\\"message\\\":\\\"Field Validation Error\\\",\\\"details\\\":\\\"Missing mandatory field dbc.\\\",\\\"type\\\":\\\"REQUEST_ERROR\\\",\\\"field\\\":\\\"dbc\\\"}]\"","time":"2019-07-26T10:57:07.500Z","v":0}]

filds-output content

[ {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "","value": "","short": false}, {"title": "There are more...","value": "There are 348 issues need to take necessary actions immediately.","short": false}
]

error output

parse error: Expected separator between values at line 1, column 254
parse error: Expected separator between values at line 1, column 254
parse error: Expected separator between values at line 1, column 245
parse error: Expected separator between values at line 1, column 245

Answer 1

Your data source is a well-formed JSON document, there is no issue at this level.

The issue occurs when you read i each line output by your jq command. As there are escaped characters, the read command will interpret them and remove them. Then, later inside your loop, the subsequent call to jq will consider that there are unescaped characters.

So I fix your script to take these two points into account:

generate_readable_output() {
        mkdir -p smoke-test-logs/tmp

        local counter=1
        local error_length=$(jq length smoke-test-logs/error-log.json)

        echo "[" > smoke-test-logs/tmp/filds-output

        jq -c '.[]' smoke-test-logs/error-log.json | while read -r i
        do
                local msg=$(echo $i | jq '.msg')
                local type=$(echo $i | jq '.name')

                echo "{\"title\": $type,\"value\": $msg,\"short\": false}," >> smoke-test-logs/tmp/filds-output

                if [ $counter -eq 20 ]
                then
                        break
                fi

                counter=$(expr $counter + 1)
        done

        if [ $error_length -gt 20 ]
        then
                echo "{\"title\": \"There are more...\",\"value\": \"There are $error_length issues need to take necessary actions immediately.\",\"short\": false}," >> smoke-test-logs/tmp/filds-output
        fi

        echo $(sed '$ s/.$//' smoke-test-logs/tmp/filds-output) > smoke-test-logs/tmp/filds-output
        echo "]" >> smoke-test-logs/tmp/filds-output
}

You can see that read is called with thr -r flag to:

-r do not allow backslashes to escape any characters