CODEWITHSUNDEEP
node.jsmemorymemory-management
user3909192
user3909192

Reputation:

How can I limit the size of WriteStream buffer in NodeJS?

I'm using a WriteStream in NodeJS to write several GB of data, and I've identified the write loop as eating up ~2GB of virtual memory during runtime (which is the GC'd about 30 seconds after the loop finishes). I'm wondering how I can limit the size of the buffer node is using when writing the stream so that Node doesn't use up so much memory during that part of the code.

I've reduced it to this trivial loop:

let ofd = fs.openSync(fn, 'w')
let ws = fs.createWriteStream('', { fd: ofd })
:
while { /*..write ~4GB of binary formatted 32bit floats and uint32s...*/ }
:
:
ws.end()

Upvotes: 1

Views: 1063

Answers (1)

Daniel Sun
Daniel Sun

Reputation: 90

The stream.write function will return a boolean value which indicate if the internal buffer is full. The buffer size is controlled by the option highWaterMark. However, this option is a threshold instead of a hard limitation, which means you can still call stream.write even if the internal buffer is full, and the memory will be used continuously if you code like this.

while (foo) {
    ws.write(bar);
}

In order to solve this issue, you have to handle the returned value false from the ws.write and waiting until the drain event of this stream is called like the following example.

async function write() {
    while (foo) {
        if (!ws.write(bar)) {
            await new Promise(resolve => ws.once('drain', resolve));
        }
    }
}

Upvotes: 1

Related Questions