CODEWITHSUNDEEP
javascriptnode.jsstream
n370
n370

Reputation: 3637

Creating a Node.js Readable stream from a JavaScript object — The simplest possible way

Can anyone explain why I'm getting an error after running the following code with node v0.10.21?

I'm aware that I could JSON.stringify() that object in order to achieve more or less the same result, the point here is to make sense of stream.Readable when objectMode is set true.

The error:

net.js:612
    throw new TypeError('invalid data');
          ^
TypeError: invalid data
    at WriteStream.Socket.write (net.js:612:11)
    at write (_stream_readable.js:583:24)
    at flow (_stream_readable.js:592:7)
    at _stream_readable.js:560:7
    at process._tickCallback (node.js:415:13)
    at Function.Module.runMain (module.js:499:11)
    at startup (node.js:119:16)
    at node.js:901:3

The code:

var stream = require('stream');
var edad = require(__dirname + '/public/edad.json');

var rs = new stream.Readable({ objectMode: true });

rs.push(edad);
rs.push(null);
rs.pipe(process.stdout);

Upvotes: 24

Views: 38694

Answers (3)

Petr Cibulka
Petr Cibulka

Reputation: 2513

I am a person who does not know much about Node (I just need to have my Gulp setup working). Because of that, the fact that I need 20+ lines of code and weird calls like stream.push(null) to work with variable in the stream seems a little crazy.

This worked for me:

var data = { "prop1": "val1", "prop2": "val2" };
var dest = 'path/to/dest/config.json';
var s = require('stream');

var stream = new s.Readable();
stream.push(JSON.stringify(data));    // stream apparently does not accept objects
stream.push(null);                    // no idea why this is needed, but it is

return stream
    .pipe()                           // whatever you need
    .pipe($.fs.createWriteStream(dest));

Upvotes: 3

slang
slang

Reputation: 640

As loganfsmyth noted, rs is in objectMode, while process.stdout isn't. So it is expecting to have Buffers/Strings written to it, and is throwing a TypeError when it gets an Object.

We need to convert the stream of Objects into text, which is what JSONStream is made to do:

JSONStream = require('JSONStream');
rs.pipe(JSONStream.stringify()).pipe(process.stdout);

Upvotes: 4

loganfsmyth
loganfsmyth

Reputation: 161627

rs is an objectMode stream, but process.stdout is not, so it is expecting to have Buffer instances written into it. Since it is getting the wrong data type, it is throwing an error.

If you wanted to be able to pipe the objects like this, you would need to have an in-between stream that supports writing as objects and reading as Buffers.

Something like this:

var stream = require('stream');
var util = require('util');

function StringifyStream(){
    stream.Transform.call(this);

    this._readableState.objectMode = false;
    this._writableState.objectMode = true;
}
util.inherits(StringifyStream, stream.Transform);

StringifyStream.prototype._transform = function(obj, encoding, cb){
    this.push(JSON.stringify(obj));
    cb();
};


var edad = require(__dirname + '/public/edad.json');

var rs = new stream.Readable({ objectMode: true });
rs.push(edad);
rs.push(null);

rs.pipe(new StringifyStream()).pipe(process.stdout);

Upvotes: 29

Related Questions