JasonYu
Reputation: 61
java process shows huge RSS consumption than expect in aarch64
Recently I deployed a java process in aarch64, but found the RSS consumption is much more higher than deploy it on x86 environment. From the jvm args, the process can only allocate for about 7 GiB memory including heap and native, but the RSS of process reached for about 10GiB, and got killed by oom-killer.
Here is an example, from /proc/$PID/status found VmRSS is 8511424KB, but from java Native Memory Tracing, it committed 2389438KB at the same time, where were this huge consumption from? How to troubleshoot further?
Here is some detailed information.
- jvm : Hotspot openjdk version "11.0.9.1" 2020-11-04
- jvm args : {-XX:+ExplicitGCInvokesConcurrent;-XX:+UseG1GC;-XX:-UseBiasedLocking;-XX:-G1UseAdaptiveIHOP;-XX:InitiatingHeapOccupancyPercent=15;-XX:G1HeapRegionSize=4M;-XX:+ExitOnOutOfMemoryError;-Djdk.nio.maxCachedBufferSize=1048576;-XX:MaxDirectMemorySize=1024m;-Xmx1024m;-Xms1024m;}
- using jemalloc-5.3.0 as memory allocator
- arm : Linux iZ2zeia8go4ym9efunf4tqZ 4.18.0-348.7.1.el8_5.aarch64
- os : CentOS8.4.2105
----------
Using strace to trace system calls of JVM Finalizer thread, the result shows that Finalizer allocated huge size memory(1G) in arm than x86(8M) at the beginning of thread start.
arm-strace-log-of-jvm-Finalizer-thread:
10:41:39.722125 set_robust_list(0xffffa053da50, 24) = 0
10:41:39.722211 mmap(NULL, 536870912, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xfffd20000000
10:41:39.722298 mmap(NULL, 536870912, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xfffd00000000
10:41:39.722330 sched_getaffinity(111949, 32, [0, 1, 2, 3]) = 8
10:41:39.722367 sched_getaffinity(111949, 32, [0, 1, 2, 3]) = 8
10:41:39.722397 getpid() = 111940
10:41:39.722421 gettid() = 111949
10:41:39.722443 rt_sigprocmask(SIG_BLOCK, NULL, [QUIT], 8) = 0
10:41:39.722472 rt_sigprocmask(SIG_UNBLOCK, [HUP INT ILL BUS FPE SEGV USR2 TERM], NULL, 8) = 0
10:41:39.722501 rt_sigprocmask(SIG_BLOCK, [QUIT], NULL, 8) = 0
10:41:39.722529 futex(0xffff201a0780, FUTEX_WAKE_PRIVATE, 1) = 1
10:41:39.722558 futex(0xffff201a4780, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
10:41:39.722631 futex(0xffff201a4728, FUTEX_WAKE_PRIVATE, 1) = 0
10:41:39.722663 mmap(0xffffa0340000, 196608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffffa0340000
10:41:39.722704 mprotect(0xffffa0340000, 196608, PROT_NONE) = 0
10:41:39.722741 prctl(PR_SET_NAME, "Finalizer") = 0
10:41:39.722807 pread64(5, "\312\376\272\276\0\0\0007\1\32\n\0<\0\260\t\0\4\0\261\t\0\4\0\262\7\0\263\7\0\264\7"..., 6080, 51205510) = 6080
10:41:39.722969 futex(0xffff201a4380, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
10:41:39.733568 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
10:41:39.733620 futex(0xffff201a4384, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
10:41:41.179815 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
10:41:41.179886 futex(0xffff201a4380, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
10:41:41.760317 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
10:41:41.760372 futex(0xffff201a4384, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
10:41:42.773671 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
10:41:42.776444 futex(0xffff201a4380, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
10:41:43.885320 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
10:41:43.885380 futex(0xffff201a4384, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
10:41:55.499245 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
10:41:55.500216 futex(0xffff201a5780, FUTEX_WAKE_PRIVATE, 1) = 1
10:41:55.500712 futex(0xffff201a4380, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
11:08:17.920555 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
11:08:17.920755 futex(0xffff201a4384, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=999880}) = -1 EAGAIN (Resource temporarily unavailable)
11:08:17.920815 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
11:08:17.928742 futex(0xffff201a4380, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
12:08:17.944661 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
12:08:17.947491 futex(0xffff201a5f80, FUTEX_WAKE_PRIVATE, 1) = 1
12:08:17.949910 futex(0xffff201a4384, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=999120}) = 0
12:08:17.950087 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
12:08:17.950171 futex(0xffff201a4380, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
12:08:17.950258 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
12:08:17.950356 futex(0xffff201a4384, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=999760}) = -1 EAGAIN (Resource temporarily unavailable)
12:08:17.950443 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
12:08:17.950511 futex(0xffff201a4380, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
12:08:17.950594 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
12:08:17.950674 futex(0xffff201a4384, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=999880}) = -1 EAGAIN (Resource temporarily unavailable)
12:08:17.950756 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
12:08:17.950824 futex(0xffff201a4380, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
12:08:17.950894 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
12:08:17.950961 futex(0xffff201a4384, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
12:08:17.951021 futex(0xffff201a4328, FUTEX_WAKE_PRIVATE, 1) = 0
x86-strace-log-of-jvm-Finalizer-thread:
10:07:40.586689 set_robust_list(0x7efca55ff9e0, 24) = 0
10:07:40.586764 mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7efca52ff000
10:07:40.586832 munmap(0x7efca52ff000, 2097152) = 0
10:07:40.586884 mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7efca5100000
10:07:40.586932 munmap(0x7efca5100000, 1048576) = 0
10:07:40.586983 munmap(0x7efca5400000, 1044480) = 0
10:07:40.587460 mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7efca5000000
10:07:40.587509 sched_getaffinity(7573, 32, [0, 1, 2, 3]) = 32
10:07:40.587957 sched_getaffinity(7573, 32, [0, 1, 2, 3]) = 32
10:07:40.588011 gettid() = 7573
10:07:40.588062 rt_sigprocmask(SIG_BLOCK, NULL, [QUIT], 8) = 0
10:07:40.588116 rt_sigprocmask(SIG_UNBLOCK, [HUP INT ILL BUS FPE SEGV USR2 TERM], NULL, 8) = 0
10:07:40.588164 rt_sigprocmask(SIG_BLOCK, [QUIT], NULL, 8) = 0
10:07:40.588214 futex(0x7efcc529c854, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7efcc529c850, FUTEX_OP_SET<<28|0<<12|FUTEX_OP_CMP_GT<<24|0x1) = 1
10:07:40.588265 futex(0x7efcc52a0854, FUTEX_WAIT_PRIVATE, 1, NULL) = 0
10:07:40.588393 futex(0x7efcc52a0828, FUTEX_WAKE_PRIVATE, 1) = 0
10:07:40.588435 mmap(0x7efca54ff000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7efca54ff000
10:07:40.588484 mprotect(0x7efca54ff000, 16384, PROT_NONE) = 0
10:07:40.588529 prctl(PR_SET_NAME, "Finalizer") = 0
10:07:40.588645 pread64(4, "\312\376\272\276\0\0\0007\1\32\n\0<\0\260\t\0\4\0\261\t\0\4\0\262\7\0\263\7\0\264\7"..., 6080, 52275686) = 6080
10:07:40.588850 futex(0x7efcc52a0554, FUTEX_WAIT_PRIVATE, 1, NULL) = 0
10:07:40.601446 futex(0x7efcc52a0528, FUTEX_WAKE_PRIVATE, 1) = 0
10:07:40.601511 futex(0x7efcc52a0554, FUTEX_WAIT_PRIVATE, 3, NULL) = 0
10:07:42.354182 futex(0x7efcc52a0528, FUTEX_WAKE_PRIVATE, 1) = 0
10:07:42.354292 futex(0x7efcc52a0554, FUTEX_WAIT_PRIVATE, 5, NULL) = 0
10:07:43.080472 futex(0x7efcc52a0528, FUTEX_WAKE_PRIVATE, 1) = 0
10:07:43.080599 futex(0x7efcc52a0554, FUTEX_WAIT_PRIVATE, 7, NULL) = 0
10:07:44.289658 futex(0x7efcc52a0528, FUTEX_WAKE_PRIVATE, 1) = 0
10:07:44.289730 futex(0x7efcc52a0554, FUTEX_WAIT_PRIVATE, 9, NULL) = 0
10:07:44.289864 futex(0x7efcc52a0528, FUTEX_WAKE_PRIVATE, 1) = 0
10:07:44.289943 futex(0x7efcc52a0554, FUTEX_WAIT_PRIVATE, 11, NULL) = 0
10:07:45.617260 futex(0x7efcc52a0528, FUTEX_WAKE_PRIVATE, 1) = 0
10:07:45.618040 futex(0x7efcc52a0554, FUTEX_WAIT_PRIVATE, 13, NULL) = 0
10:27:49.668478 futex(0x7efcc52a0528, FUTEX_WAKE_PRIVATE, 1) = 0
10:27:49.668599 futex(0x7efcc52a1854, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7efcc52a1850, FUTEX_OP_SET<<28|0<<12|FUTEX_OP_CMP_GT<<24|0x1) = 1
10:27:49.668811 futex(0x7efcc52a0554, FUTEX_WAIT_PRIVATE, 15, NULL
Upvotes: 1
Views: 194
Answers (0)
Related Questions
- Growing resident memory usage (RSS) of Java Process
- High Java memory usage even for small programs
- RSS memory growing on idle Java processes
- Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable?
- Troubleshooting unbounded Java Resident Set Size(RSS) growth
- Why my java process used memory(Linux RES) keep increasing?
- Native memory usage in Linux seems to be much higher than observed through JVM itself (e.g. through JConsole)
- Linux RHEL6 allocates more memory to run java program
- Java Process taking more and more memory
- Java process memory usage much more than what Runtime.totalMemory et al methods are telling