CODEWITHSUNDEEP
node.jsbufferreadfilestreamreadernode-streams
delux
delux

Reputation: 1886

NodeJS: How to read and modify buffer data, before parsing it from file?

In NodeJS, I am having a log file, where my logs are in the format:

{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},

These are basically objects, separated with comma. What I need to do is to read the content of this file and to convert the logs into an array of objects (that I can manipulate later).

I am trying with something like:

    let fileLogs = "./data/myfile.log";
    fs.readFile(fileLogs, (err, fileLogsContent) => {
      if (err) {
        console.log("cannot read log file");
        return;
      }

      //I know I need to manipulate the fileLogsContent here, before doing JSON.parse

      let logsContent = { ...JSON.parse(fileLogsContent) };
      //do something here with the array of objects 'logsContent'
    });

Since the content in the log file is not in a format that can be parsed, the above JSON.parse fails. My idea is to bring the logfile in the following format:

[
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}}
]

Which means that on the fly I would need to prepend [ as a first character and to replace the last , with ]. I am not aware how can I do that, since the fileLogsContent is actually a buffer. So how can I read the content and do the manipulations I mentioned, in order to be able to parse it later and to bring it into array of objects format?

Upvotes: 1

Views: 2827

Answers (2)

delux
delux

Reputation: 1886

Here is the working solution that I came up with, following the directions from the @eol's answer.

const { once } = require('events');
const { createReadStream } = require('fs');
const { createInterface } = require('readline');

(async function processLineByLine() {
  try {
    const rl = createInterface({
      input: createReadStream('./data/myfile.log'),
      crlfDelay: Infinity
    });
    const chunks = [];

    rl.on('line', (line) => {
      // Process the line.
        chunks.push(JSON.parse(`${line.substr(0, line.lastIndexOf(','))}`));
    });

    await once(rl, 'close');

    console.log('File processed. Content = ', chunks);
  } catch (err) {
    console.log("cannot read log file, err = ", err);
  }
})();

Upvotes: 0

eol
eol

Reputation: 24565

You can simply wrap each line in a string and then call JSON.parse on it after removing the trailing comma. Here's an example (note that it still needs error-handling ofc):

const fs = require('fs');
const readline = require('readline');
const readInterface = readline.createInterface({
    input: fs.createReadStream('./input.txt'),
    output: undefined,
    console: false
});

(async () => {
    const resultArray = await new Promise((resolve, reject) => {
        const chunks = [];
        readInterface.on('line', (line) => {
            line = line.substr(0, line.lastIndexOf(','))
            chunks.push(JSON.parse(`${line}`));
        })

        readInterface.on('close', () => {
            resolve(chunks);
        })
    });
    console.log(resultArray); 
})();

Upvotes: 1

Related Questions