CODEWITHSUNDEEP
jsonnode.jsoptimization
arbyter
arbyter

Reputation: 549

Efficient way to append JSON-object without parse it

I wrote a small script listening on an udp-port and stores all incoming messages (one JSON object) inside a single file. The empty file contains an array in JSON format. I'm looking for an efficient way to store all (concurrently) incoming messages from multiple clients inside this single file. The files size can be multiple hundred of megabytes large. Parsing the file and appending the new object wouldn't be efficient as needed.

Do you have an approach?

EDIT My solution, based on @t-j-crowder approach:

var dgram = require("dgram");
var fs = require("fs");
var udp_server = dgram.createSocket("udp4");

var udp_server_port = 5000

udp_server.on("message", function (msg, rinfo) {

      var json_part = "{\"message\": " + msg + "}";

        fs.open('./data/stats.json','r+',function(err,fd){
            if(err) throw err

            fs.fstat(fd,function(err,stats){
                if(err) throw err               

                if(stats.size>2){
                    json_part = new Buffer(','+json_part+']','utf-8');
                    var pos = parseInt(stats.size)-1;
                }else{
                    json_part = new Buffer('['+json_part+']','utf-8');
                    var pos = 0;
                }

                fs.write(fd,json_part, 0, json_part.length, pos, function(err,written,buffer){
                    if(err) throw err
                    fs.close(fd,function(){
                    });
                });

            });

        });

});

udp_server.bind(udp_server_port);

Regards, Marcus

Upvotes: 0

Views: 1305

Answers (1)

T.J. Crowder
T.J. Crowder

Reputation: 1075059

Fundamentally, you'll need to:

  1. Open the file using a seekable, writable stream.

  2. Seek to the end of it.

  3. Back up one character (over the closing ] of the array).

  4. Write out a comma (if this isn't the first entry) and the JSON of your new entry.

  5. Write a closing ].

  6. Close the file.

Looking at the NodeJS docs, it looks like Steps 2-4 (and arguably 5) are done all together, using the position argument of fs.write. (Be sure you open the file using r+, not one of the "append" modes.)

Upvotes: 3

Related Questions