CODEWITHSUNDEEP
node.js
ThePizzle
ThePizzle

Reputation: 1126

Why don't Node.js ReadableStreams support reading from asynchronous sources?

I'm implementing a Readable Stream. In my _read() implementation, the source of the stream is a web service which requires asynchronous calls. Why doesn't _read() provide a done callback function that can be called when my asynchronous call returns?

The Transform stream and the Writable stream both support this. Why doesn't Readable? Am I just using Readable streams improperly?

MyReadStream.prototype._read = function() {

  doSomethingAsync('foo', function(err, result) {
     if (result) {
       this.push(result);
     } else {
       this.push(null);
     }

     // why no done() available to call like in _write()?
     // done();
  }
}

In my actual implementation, I don't want to call doSomethingAsync again until a previous call has returned. Without a done callback for me to use, I have to implement my own throttle mechanism.

Upvotes: 1

Views: 192

Answers (1)

mscdex
mscdex

Reputation: 106696

_read() is a notification that the amount of buffered data is below the highWaterMark, so more data can be pulled from upstream.

_write() has a callback because it has to know when you're done processing the chunk. If you don't execute the callback for a long time, the highWaterMark may be reached and data should stop flowing in. When you execute the callback, the internal buffer can start to drain again, allowing more writes to continue.

So _read() doesn't need a callback because it's an advisory thing that you're free to ignore because it's just telling you the stream is able to buffer more data internally, whereas the callback in _write() is critical because it controls backpressure and buffering. If you need to throttle your web API calls, you may look into what the async module has to offer, especially async.memoize and/or async.queue.

Upvotes: 2

Related Questions