CODEWITHSUNDEEP
node.jsnodejs-stream
leszczu450
leszczu450

Reputation: 283

NodeJS readable streams

I have this case that when having a stream created via fs.createReadStream and using readable event and read(SIZE) to consume it my stream from time to time somehow get stucked. Sometimes there are some more bytes in stream.readableLength but the readable events are not emitted and I can not read stream entirely. What am I doing wrong?

Example:

const myStream = fs.createReadStream('my-file', { highWaterMark = 1024 });

myStream.on('readable', () => {
  const chunk = myStream.read(1500);

  if (chunk !== null) {
    // DO SOME SYNC OR ASYNC ACTIONS
  }
})

Upvotes: 0

Views: 2046

Answers (1)

jfriend00
jfriend00

Reputation: 708046

In general, using the data event and letting the stream deliver all the data to you that way is simpler and less likely to run into problems. That would be my recommendation unless there's a very specific reason you need to use the readable event and read the data yourself.

When you use the readable event and myStream.read(), you have to read everything there is until you get null back from the read indicating the buffer is empty now. Otherwise, you will get stuck waiting for the next readable event that will not come. From the doc for .read(), it says this:

The while loop is necessary when processing data with readable.read(). Only after readable.read() returns null, 'readable' will be emitted.

Here's a simple demonstration. Just point this at a large file (say 100k):

const fs = require('fs');

let stream = fs.createReadStream("vlc-help.txt");

stream.on('readable', () => {
    let data = stream.read(10)
    console.log(data);
});

When I run this with it pointing at a 343,080 byte file, I see this output:

<Buffer ef bb bf 55 73 61 67 65 3a 20>
<Buffer 76 6c 63 20 5b 6f 70 74 69 6f>

I only get two readable events and then it is stalled.

When I put in the recommended while() loop like this:

const fs = require('fs');

let stream = fs.createReadStream("vlc-help.txt");

stream.on('readable', () => {
    let data;
    while (data = stream.read(10)) {
        console.log(data);
    }
});

I get a long stream of data until the entire file is read.

Upvotes: 1

Related Questions