CODEWITHSUNDEEP
node.jsstackout-of-memory
Ritesh Sangwan
Ritesh Sangwan

Reputation: 147

node process out of memory

I want to understand why a simple node process run out of memory.

Out of fun, I tried to console.log 10 million times, after some time I got an error

<--- Last few GCs --->

  215529 ms: Scavenge 1485.5 (1532.7) -> 1485.5 (1532.7) MB, 43.9 / 0 ms (+ 1.8 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  216476 ms: Mark-sweep 1485.5 (1532.7) -> 1435.3 (1482.5) MB, 947.9 / 0 ms (+ 2.9 ms in 2 steps since start of marking, biggest step 1.8 ms) [last resort gc].
  217537 ms: Mark-sweep 1435.3 (1482.5) -> 1435.3 (1482.5) MB, 1060.2 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x76575fe3ac1 <JS Object>
    1: _writeGeneric [net.js:645] [pc=0x1f0bf7164345] (this=0x6175737bfe9 <a WriteStream with map 0x3dda56221639>,writev=0x76575f04299 <false>,data=0x3d61a01691b9 <String[18]\: madan chutiya hai\n>,encoding=0x76575fed8d9 <String[4]: utf8>,cb=0x61757359889 <JS Function WritableState.onwrite (SharedFunctionInfo 0x61757359411)>)
    2: writeOrBuffer(aka writeOrBuffer) [_stream_writable.js:~255] [pc...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6

The simple program I am executing is

for (var index = 0; index < 10000000; index++) {
        console.log('hello world');
}

What is the reason behind out of memory error?

Upvotes: 2

Views: 1806

Answers (2)

Alexander Mills
Alexander Mills

Reputation: 100010

You could also use the --expose-gc Node.js flag like so:

https://simonmcmanus.wordpress.com/2013/01/03/forcing-garbage-collection-with-node-js-and-v8/

and then use

global.gc();

within the loop; try that in combination with process.nextTick()

Upvotes: 2

Ryan
Ryan

Reputation: 5973

Update:

Ben Noordhuis talks about console.log memory here: https://groups.google.com/forum/#!topic/nodejs/KtONbpVV68U

"Each console.log() statement allocates some memory that is not reclaimed until the next tick of the event loop."

So that is why you run out of memory even though you aren't necessarily creating a bunch of objects.

Original:

Building on @Vasil's comment I think he is right. The event loop is blocked and cannot run garbage collection. FWIW I didn't have any issues running your example code on node v6.2.2. But if you play around with the allocated memory I can reproduce your issue by adding these flags: --max-executable-size=192 --max-old-space-size=256 --max-semi-space-size=2. To prove the process.nextTick() theory we can modify your test script to something like this:

// test.js
var BATCH_LIMIT = 250000;
var TEN_MILLION = 10000000;

var useNextTick = true; // switch to false to reproduce out of memory error

function log(start, end) {
  for(var index = start || 0; index<end; index++) {
    console.log('hello world', index);
    // if useNextTick is enabled we will call next tick every 250,000 iterations so GC can run
    if(useNextTick && index && index % BATCH_LIMIT === 0) {
      console.log('NEXT TICK');
      // bind the next start and end points to the log function that process.nextTick will call
      return process.nextTick(log.bind(this, index+1, index + (TEN_MILLION - index)));
    }
  }
}

// kick it off with 10,000,000 iterations
log(0, TEN_MILLION);

And run it with limited memory:

node --max-executable-size=192 --max-old-space-size=256 --max-semi-space-size=2 test.js

There is probably a cleaner way to code that but hopefully it gives you an idea of what is actually happening and why its important not to block the event loop.

Upvotes: 4

Related Questions