CODEWITHSUNDEEP
node.js
gordyr
gordyr

Reputation: 6276

How to append to a stringified JSON array using fs.createWriteStream() with Node.js

I need to write chat logs and for a variety of reasons would like to save the data as JSON.

At any given time I may have many rooms open and am currently opening a writeStream when the first user enters the room then ending the stream when the last user exits.

This writeStream object is temporarily stored in an array along with other details about the room and the users that are currently online. With each room object available at any given time like this, it is my understanding that it should be very quick and efficient to continuously write to many chat log files as the users in the various rooms chat.

I am quite new to node.js so please feel free to correct me and offer a more efficient method that above.

But onto my actual question.

Each write is a single javascript object containing specific userdata, the body of the chat and of course the time it occured.

I am currently creating txt files, and using JSON.stringify() to append a stringified version of each chat object to the file as it comes in.

This of course will yield invalid JSON resulting in a textual representation of the following:

{"userid":"1","body":"hello world"}{"userid":"2","body":"hello world 2"}{"userid":"3","body":"hello world 3"}

When in reality it needs to read :

[{"userid":"1","body":"hello world"},{"userid":"2","body":"hello world 2"},{"userid":"3","body":"hello world 3"}]

Now an obvious solution to this would be to actually open the file in node.js, parse the JSON, push the object into the array and then write all the data back. But this feels very expensive to both IO and processing-wise to me.

Another alternative would be to add a special character such as | to the beginning of each object. Then when reading the log files client side simply split the file contents at that character and JSON.parse each object in the resulting array. But surely there is a better way?

Therefore what I am looking for is a more efficient method that could deal with the logging of many chatrooms at once. Taking advantage of writeStreams if possible.

Ideally I really would like to to simply write data in the following fashion :

',' + chatobject + ']'

and then for the next write operation simply trim the end of the file by removing the ']' character and writing the next one. Leaving me with a constantly valid JSON string in the txt file.

Alternatively, is there a better method by actually writing to a .json file itself and avoid parsing altogether.

Thanks in advance, I apologize for my naivety with regards to node.js.

Upvotes: 2

Views: 1410

Answers (2)

Andbdrew
Andbdrew

Reputation: 11895

I think you are looking for something like bunyan. It lets you do stuff like this (from the docs):

var bunyan = require('bunyan');
var log = bunyan.createLogger({name: "myapp"});
log.info("hi");

Another option is to just write regular text logs separated by line instead of writing JSON. This way you can view or analyze the logs easily using regular unix-y text processing tools.

Upvotes: 2

kentcdodds
kentcdodds

Reputation: 29031

Edit: I realize this doesn't exactly answer the question, but because you said that you're new to NodeJS I thought you may not know that this kind of thing is an option. So I'm giving this as an answer as a better way to accomplish what you're trying to accomplish.

Why don't you use MongoDB through MongooseJS. It's probably a million times more of each of these things:

  1. Fast
  2. Easy
  3. Scalable
  4. Better

I would definitely recommend doing it this way. And while you're at it, you may as well use the entire MEAN stack.

Upvotes: 2

Related Questions