JRE insufficient memory issue even though with enough memory?

Question

I was running a JAVA program on a linux OS server and trying to push some information to RabbitMQ. Somehow the program was interrupted because of an Out of Memory Error. I'm trying to investigate more and figure out the reason, but the content in the log file is very rich and I would like to seek some help because I'm also new to Java.

I won't expect I get the answer directly here (but I will be very happy if I can), just want to get the right direction so that I won't waste too much time. Any help will be highly appreciated! Thanks in advance!

below is some valuable information

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 55574528 bytes for committing reserved memory.

#  Out of Memory Error (os_linux.cpp:2795), pid=43653, tid=0x00007f46186b4700
#
# JRE version: OpenJDK Runtime Environment (8.0_332-b09) (build 1.8.0_332-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.332-b09 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x00007f46280ea000):  VMThread [stack: 0x00007f46185b5000,0x00007f46186b5000] [id=43657]

Stack: [0x00007f46185b5000,0x00007f46186b5000],  sp=0x00007f46186b3330,  free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xb65ecd]  VMError::report_and_die()+0x1cd
V  [libjvm.so+0x5075aa]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0xaa
V  [libjvm.so+0x941588]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0xd8
V  [libjvm.so+0x938d9e]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2e
V  [libjvm.so+0x9b902b]  PSVirtualSpace::expand_by(unsigned long)+0x5b
V  [libjvm.so+0x9a90ca]  PSOldGen::expand_by(unsigned long) [clone .part.113]+0x2a
V  [libjvm.so+0x9a929b]  PSOldGen::expand(unsigned long) [clone .part.115]+0xdb
V  [libjvm.so+0x9a9be8]  PSOldGen::resize(unsigned long)+0x198
V  [libjvm.so+0x9af58c]  PSParallelCompact::invoke_no_policy(bool)+0x9dc
V  [libjvm.so+0x9b6b7d]  PSScavenge::invoke()+0x17d
V  [libjvm.so+0x968c73]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x63
V  [libjvm.so+0xb67b23]  VM_ParallelGCFailedAllocation::doit()+0x93
V  [libjvm.so+0xb6cfe5]  VM_Operation::evaluate()+0x55
V  [libjvm.so+0xb6afaa]  VMThread::evaluate_operation(VM_Operation*)+0x11a
V  [libjvm.so+0xb6b419]  VMThread::loop()+0x219
V  [libjvm.so+0xb6b8d2]  VMThread::run()+0x92
V  [libjvm.so+0x93e4f2]  java_start(Thread*)+0xf2

VM_Operation (0x00007f463172d3e0): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x00007f4628063800

What I see here is I get enough free memory to allocate (MemFree), right?

---------------  S Y S T E M  ---------------

OS:CentOS Linux release 7.8.2003 (Core)

uname:Linux 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022 x86_64
libc:glibc 2.17 NPTL 2.17 
rlimit: STACK 8192k, CORE 0k, NPROC 4096, NOFILE 4096, AS infinity
load average:4.81 6.40 3.68

/proc/meminfo:
MemTotal:        3861256 kB
MemFree:          108460 kB
MemAvailable:      19104 kB
Buffers:               0 kB
Cached:            22388 kB
SwapCached:         7600 kB
Active:          2577112 kB
Inactive:         971876 kB
Active(anon):    2570780 kB
Inactive(anon):   957160 kB
Active(file):       6332 kB
Inactive(file):    14716 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1048572 kB
SwapFree:            780 kB
Dirty:                20 kB
Writeback:             0 kB
AnonPages:       3519584 kB
Mapped:             3356 kB
Shmem:              1184 kB
Slab:              85128 kB
SReclaimable:      40496 kB
SUnreclaim:        44632 kB
KernelStack:        7872 kB
PageTables:        20908 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2979200 kB
Committed_AS:    3444760 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      183428 kB
VmallocChunk:   34359310332 kB
Percpu:            24064 kB
HardwareCorrupted:     0 kB
AnonHugePages:    448512 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       94016 kB
DirectMap2M:     4100096 kB
DirectMap1G:     2097152 kB

Answer 1

What I see here is I get enough free memory to allocate (MemFree), right?

No, an application cannot utilize all physical memory; Linux reserves a certain amount of free RAM exclusively for kernel use. This can be configured by vm.min_free_kbytes sysctl option, the default is typically 44-66 MB.

$ sudo sysctl vm.min_free_kbytes
vm.min_free_kbytes = 45056

Even if there is enough free RAM, an allocation may fail because of other restrictions, like

• vm.max_map_count
• vm.overcommit_memory, vm.overcommit_ratio
• RLIMIT_AS

See this answer for details.