Parsing a log file text into a JSON

Question

I'm trying to create a process to parse a log file into a JSON file to a future processing.

The log file has this format:

[
  {
    "textPayload": "Dialogues0 gRPC Response : response_id: \"fc4e2e63-509b4ae2-a7d8-c401e563aa4b\"\nquery_result {\n  query_text: \"wonderful\"\naction: \"smalltalk.appraisal.good\"\n  parameters {\n  }\nall_required_params_present: true\n  fulfillment_text: \"Glad you think so!\"\n  fulfillment_messages {\n    text {\n      text: \"Glad you think so!\"\n    }\n  }\n  intent_detection_confidence: 1.0\n  language_code: \"en\"\nsentiment_analysis_result {\n    query_text_sentiment {\n    }\n}\n}\nagent_id: \"3d22af45-f603-4a8a-a7ce-a9b2af47b762\"\n",
"insertId": "1lfux63g16s1nna",
"resource": {
  "type": "global",
  "labels": {
    "project_id": "data-analytics-digital-dev"
  }
},
"timestamp": "2018-11-07T14:31:02.435Z",
"severity": "INFO",
"labels": {
  "request_id": "fc4e2e63-509b-4ae2-a7d8-c401e563aa4b",
  "type": "dialogflow_response",
  "protocol": "V2BETA1"
},
"logName": "projects/data-analytics-digital-dev/logs/dialogflow_agent",
"trace": "7fa08c8c-df50-4d46-9f20-b1e357b844a4",
"receiveTimestamp": "2018-11-07T14:31:02.555590714Z"
  }
]

My target is the content of "textPayload"

the Node.js code is:

fs = require('fs');
fs.readFile('./global_logs1.json', 'utf8', function (err,data) {
  if (err) {
    return console.log(err);
  }
    let parsedLog = JSON.parse(data);
    for(let myKey in parsedLog) {
        let tempJson = (parsedLog[myKey]["textPayload"]);
        tempJson = (tempJson.substr(91,));
        tempJson = (tempJson.substr(0, tempJson.length - 50));
        console.log((tempJson));
        //console.log(JSON.parse(tempJson));
   }
});

and it is resulting in a JSON like string:

{
  query_text: "wonderful"
  action: "smalltalk.appraisal.good"
  parameters {
  }
  all_required_params_present: true
  fulfillment_text: "Glad you think so!"
  fulfillment_messages {
    text {
      text: "Glad you think so!"
    }
  }
  intent_detection_confidence: 1.0
  language_code: "en"
  sentiment_analysis_result {
    query_text_sentiment {
    }
  }
}

However when I call the JSON.parser I receive an error:

undefined:2
  query_text: "wonderful"
  ^

SyntaxError: Unexpected token q in JSON at position 5
     at JSON.parse (<anonymous>)
     at c:\Codes\Logging\test.js:15:26
     at FSReqWrap.readFileAfterClose [as oncomplete] 
(internal/fs/read_file_context.js:53:3)

Seems it is missing double quotes, but I'm not sure.

Any idea on how to do it?

Answer 1

The text in the 'textPayload' attribute is absolutely not valid JSON. As you pointed out, the attribute names are missing double quotes. You are going to need to parse it out on your own. You could experiment with regex (results and opinions may vary.) or use existing "relaxed JSON" libraries that do not make use of eval.

EDIT: Using the module 'relaxed-json' I threw together this dirty script. You obviously can sanitize it yourself instead of relying on an external module but I am lazy here, and there might even be an out of the box solution to clean this up better, but this works. You need to remove the newlines, add commas after the items, add colons to the object attributes and add double quotes to the attribute names in order for it to be valid JSON. Like I said, its a dirty script and I did some awful regex matching and replacing just for the sake of making a proof of concept, so I am prepared to be judged on how crappy it is.

var parsed = require('./payload.json');
const rjson = require('relaxed-json');

for(let key in parsed){
        let tempJson = (parsed[key]["textPayload"]);
        tempJson = (tempJson.substr(91,));
        tempJson = (tempJson.substr(0, tempJson.length - 50));
        tempJson = tempJson.replace(/\n/g,",");
        tempJson = tempJson.replace(/\{,/g,"{");
        tempJson = tempJson.replace(/ \{/g,":{");
        let transformed = JSON.parse(rjson.transform(tempJson));
        console.log(transformed);

}

The output is a true javascript object now.

{ query_text: 'wonderful',
  action: 'smalltalk.appraisal.good',
  parameters: {},
  all_required_params_present: true,
  fulfillment_text: 'Glad you think so!',
  fulfillment_messages: { text: { text: 'Glad you think so!' } },
  intent_detection_confidence: 1,
  language_code: 'en',
  sentiment_analysis_result: { query_text_sentiment: {} } }

Answer 2

Your tempJson is actually a javascript object.

To convert it back to json just call

JSON.stringify(tempJson)

Sometimes the two concepts get mixed because they are so easily interchangeable.