Buffering in Node

Question

I'm learning basic node before I delve in to frameworks such as connect and express. I've come across the subject of buffers. When do I need to use buffers in node and how do I know how big to make the buffer?

If I have a string of text being sent in a POST request do I need to buffer this?

var server = http.createServer(function(req,res) {
    var body = '';
    req.setEncoding('utf8');
    req.on('data', function(chunk) { body += chunk});
    req.on('end', function() { //... });
});

I know connect provides limit() functionality but I'm not sure if this defers to buffer in the background. With that aside I'm specifically interested to know if, in vanilla node, I need to use buffers for data such as JSON and strings or should I just use a buffer when I have data that could potentially be large like a file upload?

Answer 1

I'm not sure what you are asking for but you have 2 options here. You either retrieve all the data as string then wrap it as a buffer or concat the buffer immediately.

new Buffer

var server = http.createServer(function(req,res) {
    var body = '';
    req.setEncoding('utf8');
    req.on('data', function(chunk) { body += chunk});
    req.on('end', function() {
      var myBuffer = new Buffer(body); // this is now a buffer of the received chunks
    });
});

Buffer.concat

var server = http.createServer(function(req,res) {
    var bufs = [];
    req.on('data', function(chunk) { bufs.push(chunk); });
    req.on('end', function() {
      var myBuffer = Buffer.concat(bufs); // this is now a buffer of the received chunks
    });
});

Note

Pure JavaScript is Unicode friendly but not nice to binary data. When dealing with TCP streams or the file system, it's necessary to handle octet streams. Node has several strategies for manipulating, creating, and consuming octet streams.

Raw data is stored in instances of the Buffer class. A Buffer is similar to an array of integers but corresponds to a raw memory allocation outside the V8 heap. A Buffer cannot be resized.

The Buffer class is a global, making it very rare that one would need to ever require('buffer').

• source