CODEWITHSUNDEEP
node.jsgzipzlib
Sean256
Sean256

Reputation: 3099

Streaming data to gzip file crashes with "JavaScript heap out of memory" (NodeJS)

I have this little script to dump a bunch of text data from a source to disk in the form of gzip. Most sources I pull from work without issue, but I've come up against one which is throwing JavaScript heap out of memory.

Here's a snippet of what it's doing

const fs = require('fs');
const zlib = require('zlib');

const file = fs.createWriteStream('file.gz');
const gzip = zlib.createGzip();
gzip.pipe(file);

// ... code to connect to someDataSource would be here

someDataSource.on('data', (line) => { // feeding lines of text
    gzip.write(line);
});

someDataSource.on('done', () => {
    // crashes before this point
    gzip.end();
});

I suspect the zlib module is buffering way more than it should before flushing to disk. At the time of the crash the gz file is only about 4MB large. Like I said above other data sources I pull from work, and all of those produce gz files well over 50MB.

The docs on the module are here: https://nodejs.org/api/zlib.html#zlib_class_options

I'm not sure how to tweak the options to get this to behave.

CRASH:

<--- Last few GCs --->

[33692:0x10264e000]    97556 ms: Scavenge 1370.6 (1411.7) -> 1363.3 (1412.2) MB, 4.5 / 0.0 ms  (average mu = 0.174, current mu = 0.137) allocation failure 
[33692:0x10264e000]    97569 ms: Scavenge 1371.0 (1412.2) -> 1363.7 (1413.7) MB, 4.5 / 0.0 ms  (average mu = 0.174, current mu = 0.137) allocation failure 
[33692:0x10264e000]    97582 ms: Scavenge 1371.3 (1413.7) -> 1364.0 (1430.2) MB, 4.5 / 0.0 ms  (average mu = 0.174, current mu = 0.137) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0xdd88f3dbe3d]
Security context: 0x32b80cc1e6e9 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [0x32b897904941] [/some/path/node_modules/tedious/lib/token/stream-parser.js:~154] [pc=0xdd88f6fbec4](this=0x32b8101826f1 <undefined>)
    2: valueParse(aka valueParse) [0x32b8c73a8ab9] [/some/path/node_modules/tedious/lib/value-parser.js:~74] [pc=0xdd88f6c96d3](this=0x32b8101826f1 ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003c597 node::Abort() [/usr/local/bin/node]
 2: 0x10003c7a1 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1001ad575 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x100579242 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: 0x10057bd15 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 6: 0x100577bbf v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 7: 0x100575d94 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x100574998 v8::internal::Heap::HandleGCRequest() [/usr/local/bin/node]
 9: 0x10052a1c8 v8::internal::StackGuard::HandleInterrupts() [/usr/local/bin/node]
10: 0x1007d9bb1 v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
11: 0xdd88f3dbe3d 
12: 0xdd88f6fbec4 
13: 0xdd88f6c96d3 
14: 0xdd88f6c8870 
[1]    33692 abort      node app.js

Upvotes: 0

Views: 646

Answers (2)

user8246956
user8246956

Reputation:

You can also try to increase the memory allocated to Node.js:

node --max-old-space-size=8192 your_script.js

Upvotes: 0

lx1412
lx1412

Reputation: 1200

add a drain eventlistener. Because writing data to bootstrap is a synchronous behaviour.

someDataSource.on('data', (line) => { // feeding lines of text
    const ok = gzip.write(line);
    if(!ok) {
        someDataSource.pause();
    }
});
gzip.on('drain', () => {
    someDataSource.resume();
});

someDataSource.on('done', () => {
    // crashes before this point
    gzip.end();
});

or use pipe method directly.

someDataSource.pipe(gzip).pipe(file);

Upvotes: 2

Related Questions