FATAL ERROR: Scavenger: semi-space copy Allocation failed

Question

I'm getting an error whilst running a node application which deals with a very large file (76Million records) being read into an array.

The Suse server has 64GB of RAM and I have swap file of 100GB as well. however the heap dump happens at around 30GB of memory. with around 31M records read.

Node version is 12.22.1

This is a analytics application and its a one off workload of the initial pull of data. The application reads the files by chunks of 100k records at a time.

The Parameters I have tried multiple permutations of these and this is the last run one.

--max-old-space-size=196608
--max-semi-space-size=256
--semi-space-growth-factor=4

I have run this sort of large datasets on previously on Node 4 using only --max-old-space-size without an issue but out of what I have read the memory management is different from node 12 onwards.

Appreciate any guidance or advice you could give.

Thanks and regards,

Dilanke

^[[0;34mINFO ^[[0m      2021-04-27 08:10:50     ^[[0;1mDSParser^[[0m    Read 31700000 lines
                ^[[0;4mtable^[[0m: "ekpo"
                ^[[0;4mstep^[[0m: 0
                ^[[0;4mmemory^[[0m: {"rss":21905813504,"heapTotal":17302638592,"heapUsed":17253034680,"external":4473602959,"arrayBuffers":4472358538}
^[[0;34mINFO ^[[0m      2021-04-27 08:10:56     ^[[0;1mDSParser^[[0m    Read 31800000 lines
                ^[[0;4mtable^[[0m: "ekpo"
                ^[[0;4mstep^[[0m: 0
                ^[[0;4mmemory^[[0m: {"rss":22110760960,"heapTotal":17492168704,"heapUsed":17435101184,"external":4473635727,"arrayBuffers":4472391306}

<--- Last few GCs --->

[11070:0x4c9f3c0]  1092787 ms: Scavenge 16733.7 (16784.8) -> 16733.6 (16784.8) MB, 84.5 / 0.0 ms  (average mu = 0.988, current mu = 0.990) allocation failure
[11070:0x4c9f3c0]  1092869 ms: Scavenge 16733.7 (16784.8) -> 16733.6 (16784.8) MB, 80.9 / 0.0 ms  (average mu = 0.988, current mu = 0.990) allocation failure
[11070:0x4c9f3c0]  1092953 ms: Scavenge 16733.7 (16784.8) -> 16733.7 (16784.8) MB, 83.9 / 0.0 ms  (average mu = 0.988, current mu = 0.990) allocation failure


<--- JS stacktrace --->
Cannot get stack trace in GC.

>  FATAL ERROR: Scavenger: semi-space copy Allocation failed -
> JavaScript heap out of memory  1: 0xa1a640 node::Abort()
> [/home/forestpin/data/retail/bin/node/node/bin/node]  2: 0xa1aa4c
> node::OnFatalError(char const*, char const*)
> [/home/forestpin/data/retail/bin/node/node/bin/node]  3: 0xb9a68e
> v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool)
> [/home/forestpin/data/retail/bin/node/node/bin/node]  4: 0xb9aa09
> v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char
> const*, bool) [/home/forestpin/data/retail/bin/node/node/bin/node]  5:
> 0xd57c85  [/home/forestpin/data/retail/bin/node/node/bin/node]  6:
> 0xdba162  [/home/forestpin/data/retail/bin/node/node/bin/node]  7:
> 0xdbbfb6 void
> v8::internal::BodyDescriptorBase::IteratePointers(v8::internal::HeapObject,
> int, int, v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node]  8: 0xdbc680 void
> v8::internal::BodyDescriptorBase::IterateJSObjectBodyImpl(v8::internal::Map,
> v8::internal::HeapObject, int, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node]  9: 0xdbcfd1 void
> v8::internal::BodyDescriptorApply v8::internal::Map, v8::internal::HeapObject, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*>(v8::internal::InstanceType,
> v8::internal::Map, v8::internal::HeapObject, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 10: 0xdbd87e
> v8::internal::Scavenger::IterateAndScavengePromotedObject(v8::internal::HeapObject,
> v8::internal::Map, int)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 11: 0xdbdb8b
> v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 12: 0xdc15bd
> v8::internal::ScavengingTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 13: 0xd75885
> v8::internal::ItemParallelJob::Task::RunInternal()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 14: 0xd75c94
> v8::internal::ItemParallelJob::Run()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 15: 0xdbf65a
> v8::internal::ScavengerCollector::CollectGarbage()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 16: 0xd50741
> v8::internal::Heap::Scavenge()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 17: 0xd64f1b
> v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,
> v8::GCCallbackFlags)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 18: 0xd65a85
> v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,
> v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 19: 0xd6853c
> v8::internal::Heap::AllocateRawWithRetryOrFail(int,
> v8::internal::AllocationType, v8::internal::AllocationOrigin,
> v8::internal::AllocationAlignment)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 20: 0xd2ef5b
> v8::internal::Factory::NewFillerObject(int, bool,
> v8::internal::AllocationType, v8::internal::AllocationOrigin)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 21: 0x107158e
> v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*,
> v8::internal::Isolate*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 22: 0x140de99 
> [/home/forestpin/data/retail/bin/node/node/bin/node]

FATAL ERROR: Scavenger: semi-space copy Allocation failed

Answers (1)

Related Questions