What does this JVM crash log mean?

Question

I have a java chat server program running in linux. And sometimes the fatal error below happens. I think this fatal error may be caused by OOM but I am not sure.

-------------------- crash log ---------------------------

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (sharedRuntime.cpp:834), pid=17781, tid=140031174805248
#  fatal error: exception happened outside interpreter, nmethods and vtable stubs at pc 0x00007f5c1d05406f
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f5bac160800):  JavaThread "nioEventLoopGroup-3-12" [_thread_in_Java, id=18017, stack(0x00007f5b8c5e8000,0x00007f5b8c6e9000)]

Stack: [0x00007f5b8c5e8000,0x00007f5b8c6e9000],  sp=0x00007f5b8c6e6150,  free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xaac99a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x4f2de9]  report_fatal(char const*, int, char const*)+0x59
V  [libjvm.so+0x9ab5ba]  SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned char*, SharedRuntime::ImplicitExceptionKind)+0x33a
V  [libjvm.so+0x914f1a]  JVM_handle_linux_signal+0x48a
V  [libjvm.so+0x90b493]  signalHandler(int, siginfo*, void*)+0x43
C  [libpthread.so.0+0xf100]
J 10415 C2 com.sun.crypto.provider.GCTR.doFinal([BII[BI)I (130 bytes) @ 0x00007f5c1e10c096 [0x00007f5c1e10be40+0x256]


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f5bb00bb800 JavaThread "OkHttp Dispatcher" [_thread_blocked, id=16894, stack(0x00007f5bbd8c9000,0x00007f5bbd9ca000)]
  0x00007f5bc81cb000 JavaThread "RMI TCP Connection(38403)-192.241.201.147" daemon [_thread_in_native, id=16892, stack(0x00007f5b7d0b1000,0x00007f5b7d1b2000)]
  0x00007f5bb800e000 JavaThread "RMI TCP Connection(38400)-165.227.2.145" daemon [_thread_in_native, id=16891, stack(0x00007f5b8e4f4000,0x00007f5b8e5f5000)]
  0x00007f5bc82ba000 JavaThread "RMI TCP Connection(38404)-192.241.201.147" daemon [_thread_in_native, id=16885, stack(0x00007f5b7fedb000,0x00007f5b7ffdc000)]
  0x00007f5bc82d4000 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=16883, stack(0x00007f5b7d7b8000,0x00007f5b7d8b9000)]
  0x00007f5bb8019000 JavaThread "RMI TCP Connection(38396)-165.227.2.145" daemon [_thread_in_native, id=16882, stack(0x00007f5b808e5000,0x00007f5b809e6000)]
  0x00007f5bb00a9800 JavaThread "OkHttp Dispatcher" [_thread_blocked, id=16878, stack(0x00007f5b7d5b6000,0x00007f5b7d6b7000)]
  0x00007f5b98417800 JavaThread "OkHttp Dispatcher" [_thread_blocked, id=16877, stack(0x00007f5b8e9f9000,0x00007f5b8eafa000)]
  0x00007f5b98421000 JavaThread "OkHttp Dispatcher" [_thread_blocked, id=16876, stack(0x00007f5b805e2000,0x00007f5b806e3000)]
  0x00007f5b980fd800 JavaThread "OkHttp Dispatcher" [_thread_blocked, id=16787, stack(0x00007f5bbc6b7000,0x00007f5bbc7b8000)]
  0x00007f5bb00b3800 JavaThread "MonitoringThread-8" daemon [_thread_in_native, id=16771, stack(0x00007f5b7dabb000,0x00007f5b7dbbc000)]
  0x00007f5bb00b4800 JavaThread "MonitoringThread-7" daemon [_thread_in_native, id=16770, stack(0x00007f5b810ed000,0x00007f5b811ee000)]
  0x00007f5bb002d000 JavaThread "MonitoringThread-52" daemon [_thread_in_native, id=16763, stack(0x00007f5b8c7ea000,0x00007f5b8c8eb000)]
  0x00007f5bb001e000 JavaThread "MonitoringThread-6" daemon [_thread_in_native, id=16762, stack(0x00007f5b80ce9000,0x00007f5b80dea000)]
  0x00007f5bb0057800 JavaThread "MonitoringThread-5" daemon [_thread_in_native, id=16761, stack(0x00007f5b7ebc8000,0x00007f5b7ecc9000)]
  0x00007f5bb0098000 JavaThread "MonitoringThread-12" daemon [_thread_in_native, id=16757, stack(0x00007f5b7bfa0000,0x00007f5b7c0a1000)]
  0x00007f5bb00a8000 JavaThread "MonitoringThread-11" daemon [_thread_in_native, id=16756, stack(0x00007f5b806e3000,0x00007f5b807e4000)]
  0x00007f5bb0088800 JavaThread "MonitoringThread-12" daemon [_thread_in_native, id=16732, stack(0x00007f5b7fdda000,0x00007f5b7fedb000)]
  0x00007f5bb00c5000 JavaThread "MonitoringThread-11" daemon [_thread_in_native, id=16731, stack(0x00007f5b8e8f8000,0x00007f5b8e9f9000)]
  0x00007f5bb0096800 JavaThread "MonitoringThread-51" daemon [_thread_in_native, id=16648, stack(0x00007f5b8ecfc000,0x00007f5b8edfd000)]
  0x00007f5bb00ba800 JavaThread "MonitoringThread-5" daemon [_thread_in_native, id=16573, stack(0x00007f5b7ffdc000,0x00007f5b800dd000)]
  0x00007f5bb012a000 JavaThread "MonitoringThread-4" daemon [_thread_in_native, id=16572, stack(0x00007f5b7e1c2000,0x00007f5b7e2c3000)]
  0x00007f5bb00a3000 JavaThread "MonitoringThread-6" daemon [_thread_in_native, id=16535, stack(0x00007f5b7fcd9000,0x00007f5b7fdda000)]
  0x00007f5bb00a0800 JavaThread "MonitoringThread-5" daemon [_thread_in_native, id=16533, stack(0x00007f5b7dbbc000,0x00007f5b7dcbd000)]
  0x00007f5bb0129000 JavaThread "MonitoringThread-2" daemon [_thread_in_native, id=16374, stack(0x00007f5b8eafa000,0x00007f5b8ebfb000)]
  0x00007f5bb0128000 JavaThread "pool-2168-thread-2" [_thread_blocked, id=16373, stack(0x00007f5bbc4b5000,0x00007f5bbc5b6000)]
  0x00007f5bb0097800 JavaThread "MonitoringThread-1" daemon [_thread_in_native, id=16372, stack(0x00007f5b7f7d4000,0x00007f5b7f8d5000)]
  0x00007f5bb00f1000 JavaThread "pool-2168-thread-1" [_thread_blocked, id=16371, stack(0x00007f5b7f9d6000,0x00007f5b7fad7000)]
  0x00007f5bb00bf000 JavaThread "pool-2201-thread-2" [_thread_blocked, id=16251, stack(0x00007f5b7c5a6000,0x00007f5b7c6a7000)]
  0x00007f5bb00aa000 JavaThread "MonitoringThread-4" daemon [_thread_in_native, id=16233, stack(0x00007f5b828f4000,0x00007f5b829f5000)]
  0x00007f5bb00b7800 JavaThread "MonitoringThread-3" daemon [_thread_in_native, id=16232, stack(0x00007f5b800dd000,0x00007f5b801de000)]
  0x00007f5bb00ae800 JavaThread "pool-2174-thread-2" [_thread_blocked, id=16118, stack(0x00007f5b8c1e4000,0x00007f5b8c2e5000)]
  0x00007f5bb00a4800 JavaThread "pool-2174-thread-1" [_thread_blocked, id=16116, stack(0x00007f5bbd7c8000,0x00007f5bbd8c9000)]
  0x00007f5bb00e1000 JavaThread "pool-2201-thread-1" [_thread_blocked, id=16084, stack(0x00007f5b80fec000,0x00007f5b810ed000)]
  0x00007f5bb0108800 JavaThread "pool-2231-thread-1" [_thread_blocked, id=16082, stack(0x00007f5b815f2000,0x00007f5b816f3000)]
  0x00007f5bb00c0000 JavaThread "pool-2159-thread-2" [_thread_blocked, id=16080, stack(0x00007f5b82df9000,0x00007f5b82efa000)]
  0x00007f5bb00be000 JavaThread "pool-2159-thread-1" [_thread_blocked, id=16078, stack(0x00007f5b8c0e3000,0x00007f5b8c1e4000)]
  0x00007f5bb016a000 JavaThread "pool-2165-thread-2" [_thread_blocked, id=16064, stack(0x00007f5b8e3f3000,0x00007f5b8e4f4000)]
  0x00007f5bb00ec000 JavaThread "pool-2165-thread-1" [_thread_blocked, id=16062, stack(0x00007f5b8e7f7000,0x00007f5b8e8f8000)]
  0x00007f5bb00a7000 JavaThread "pool-2156-thread-2" [_thread_blocked, id=16004, stack(0x00007f5bbc7b8000,0x00007f5bbc8b9000)]
  0x00007f5bb0089800 JavaThread "pool-2156-thread-1" [_thread_blocked, id=16002, stack(0x00007f5bbd4c5000,0x00007f5bbd5c6000)]
  0x00007f5bb00c3800 JavaThread "pool-2171-thread-2" [_thread_blocked, id=16000, stack(0x00007f5bbd6c7000,0x00007f5bbd7c8000)]
  0x00007f5bb003a000 JavaThread "pool-2171-thread-1" [_thread_blocked, id=15998, stack(0x00007f5b7bc9d000,0x00007f5b7bd9e000)]
  0x00007f5bb00c1000 JavaThread "pool-2198-thread-2" [_thread_blocked, id=15995, stack(0x00007f5bbf5dc000,0x00007f5bbf6dd000)]
  0x00007f5bb010a800 JavaThread "pool-2180-thread-2" [_thread_blocked, id=15994, stack(0x00007f5bbf8df000,0x00007f5bbf9e0000)]
  0x00007f5bb0056000 JavaThread "pool-2198-thread-1" [_thread_blocked, id=15991, stack(0x00007f5b8c6e9000,0x00007f5b8c7ea000)]
  0x00007f5bb00ad000 JavaThread "pool-2180-thread-1" [_thread_blocked, id=15990, stack(0x00007f5b8c3e6000,0x00007f5b8c4e7000)]
  0x00007f5b98434800 JavaThread "pool-2250-thread-1" [_thread_blocked, id=15989, stack(0x00007f5bbc5b6000,0x00007f5bbc6b7000)]
  0x00007f5b98408800 JavaThread "pool-2247-thread-1" [_thread_blocked, id=15988, stack(0x00007f5bbd1c2000,0x00007f5bbd2c3000)]
  0x00007f5b9840c800 JavaThread "pool-2244-thread-1" [_thread_blocked, id=15987, stack(0x00007f5bbccbd000,0x00007f5bbcdbe000)]
  0x00007f5b98433000 JavaThread "pool-2241-thread-1" [_thread_blocked, id=15986, stack(0x00007f5b804e1000,0x00007f5b805e2000)]
  0x00007f5b98431800 JavaThread "pool-2238-thread-1" [_thread_blocked, id=15985, stack(0x00007f5bbc8b9000,0x00007f5bbc9ba000)]
  0x00007f5b98426000 JavaThread "pool-2235-thread-1" [_thread_blocked, id=15984, stack(0x00007f5b7eac7000,0x00007f5b7ebc8000)]
  0x00007f5b9840a000 JavaThread "pool-2232-thread-1" [_thread_blocked, id=15983, stack(0x00007f5b7e9c6000,0x00007f5b7eac7000)]
  0x00007f5b98438800 JavaThread "pool-2229-thread-1" [_thread_blocked, id=15982, stack(0x00007f5bbcfc0000,0x00007f5bbd0c1000)]
  0x00007f5b983fd800 JavaThread "pool-2226-thread-1" [_thread_blocked, id=15981, stack(0x00007f5b807e4000,0x00007f5b808e5000)]
  0x00007f5b98404800 JavaThread "pool-2223-thread-1" [_thread_blocked, id=15980, stack(0x00007f5b80dea000,0x00007f5b80eeb000)]
  0x00007f5b9814f000 JavaThread "pool-2220-thread-1" [_thread_blocked, id=15979, stack(0x00007f5b80ae7000,0x00007f5b80be8000)]
  0x00007f5b98436800 JavaThread "pool-2217-thread-1" [_thread_blocked, id=15978, stack(0x00007f5b7d6b7000,0x00007f5b7d7b8000)]
  0x00007f5b9825f800 JavaThread "pool-2214-thread-1" [_thread_blocked, id=15977, stack(0x00007f5b7f0cd000,0x00007f5b7f1ce000)]
  0x00007f5b98414000 JavaThread "pool-2211-thread-1" [_thread_blocked, id=15976, stack(0x00007f5b7e2c3000,0x00007f5b7e3c4000)]
  0x00007f5b9841f000 JavaThread "pool-2208-thread-1" [_thread_blocked, id=15975, stack(0x00007f5b8ebfb000,0x00007f5b8ecfc000)]
  0x00007f5b98563800 JavaThread "pool-2205-thread-1" [_thread_blocked, id=15974, stack(0x00007f5bbd3c4000,0x00007f5bbd4c5000)]
  0x00007f5b9842c000 JavaThread "pool-2202-thread-1" [_thread_blocked, id=15973, stack(0x00007f5b802df000,0x00007f5b803e0000)]
  0x00007f5b98416000 JavaThread "pool-2199-thread-1" [_thread_blocked, id=15972, stack(0x00007f5bbcebf000,0x00007f5bbcfc0000)]
  0x00007f5b9842a800 JavaThread "pool-2196-thread-1" [_thread_blocked, id=15971, stack(0x00007f5b8e5f5000,0x00007f5b8e6f6000)]
  0x00007f5b98428800 JavaThread "pool-2193-thread-1" [_thread_blocked, id=15970, stack(0x00007f5b7fbd8000,0x00007f5b7fcd9000)]
  0x00007f5b9800e000 JavaThread "pool-2190-thread-1" [_thread_blocked, id=15969, stack(0x00007f5b7f6d3000,0x00007f5b7f7d4000)]
  0x00007f5b980fd000 JavaThread "pool-2187-thread-1" [_thread_blocked, id=15968, stack(0x00007f5bbc2b3000,0x00007f5bbc3b4000)]
  0x00007f5b98564800 JavaThread "pool-2184-thread-1" [_thread_blocked, id=15967, stack(0x00007f5bbcabb000,0x00007f5bbcbbc000)]
  0x00007f5b98401000 JavaThread "pool-2181-thread-1" [_thread_blocked, id=15966, stack(0x00007f5b7b798000,0x00007f5b7b899000)]
  0x00007f5b9840e000 JavaThread "pool-2178-thread-1" [_thread_blocked, id=15965, stack(0x00007f5bbc3b4000,0x00007f5bbc4b5000)]
  0x00007f5b9842e000 JavaThread "pool-2175-thread-1" [_thread_blocked, id=15964, stack(0x00007f5b8edfd000,0x00007f5b8eefe000)]
  0x00007f5b98567000 JavaThread "pool-2172-thread-1" [_thread_blocked, id=15963, stack(0x00007f5b7ddbe000,0x00007f5b7debf000)]
  0x00007f5b98419800 JavaThread "pool-2169-thread-1" [_thread_blocked, id=15962, stack(0x00007f5bbd2c3000,0x00007f5bbd3c4000)]
  0x00007f5b98044000 JavaThread "pool-2166-thread-1" [_thread_blocked, id=15961, stack(0x00007f5b8e6f6000,0x00007f5b8e7f7000)]
  0x00007f5b98424800 JavaThread "pool-2163-thread-1" [_thread_blocked, id=15960, stack(0x00007f5b7f8d5000,0x00007f5b7f9d6000)]
  0x00007f5b98412000 JavaThread "pool-2160-thread-1" [_thread_blocked, id=15959, stack(0x00007f5b7fad7000,0x00007f5b7fbd8000)]
  0x00007f5b9842f800 JavaThread "pool-2157-thread-1" [_thread_blocked, id=15958, stack(0x00007f5b7caab000,0x00007f5b7cbac000)]
  0x00007f5b98076000 JavaThread "pool-2154-thread-1" [_thread_blocked, id=15957, stack(0x00007f5b814f1000,0x00007f5b815f2000)]
  0x00007f5b98410000 JavaThread "pool-2151-thread-1" [_thread_blocked, id=15956, stack(0x00007f5b7d3b4000,0x00007f5b7d4b5000)]
  0x00007f5b98403000 JavaThread "pool-2148-thread-1" [_thread_blocked, id=15955, stack(0x00007f5b7e4c5000,0x00007f5b7e5c6000)]
  0x00007f5bac0d6800 JavaThread "nioEventLoopGroup-3-16" [_thread_in_native, id=18174, stack(0x00007f5b813f0000,0x00007f5b814f1000)]
  0x00007f5bac0e8800 JavaThread "nioEventLoopGroup-3-15" [_thread_in_native, id=18151, stack(0x00007f5b7edca000,0x00007f5b7eecb000)]
  0x00007f5bc4618000 JavaThread "Druid-ConnectionPool-Destroy-741286713" daemon [_thread_blocked, id=18147, stack(0x00007f5b7eecb000,0x00007f5b7efcc000)]
  0x00007f5c2d149000 JavaThread "Druid-ConnectionPool-Create-741286713" daemon [_thread_blocked, id=18146, stack(0x00007f5b7efcc000,0x00007f5b7f0cd000)]
  0x00007f5bac0e6000 JavaThread "nioEventLoopGroup-3-14" [_thread_in_native, id=18070, stack(0x00007f5b80eeb000,0x00007f5b80fec000)]
  0x00007f5bac0e5800 JavaThread "nioEventLoopGroup-3-13" [_thread_in_native, id=18018, stack(0x00007f5b811ee000,0x00007f5b812ef000)]
=>0x00007f5bac160800 JavaThread "nioEventLoopGroup-3-12" [_thread_in_Java, id=18017, stack(0x00007f5b8c5e8000,0x00007f5b8c6e9000)]
  0x00007f5bac15c800 JavaThread "nioEventLoopGroup-3-11" [_thread_in_native, id=17944, stack(0x00007f5b812ef000,0x00007f5b813f0000)]
  0x00007f5bac15b800 JavaThread "nioEventLoopGroup-3-10" [_thread_in_native, id=17935, stack(0x00007f5b82bf7000,0x00007f5b82cf8000)]
  0x00007f5bac15a800 JavaThread "nioEventLoopGroup-3-9" [_thread_in_native, id=17932, stack(0x00007f5b829f5000,0x00007f5b82af6000)]
  0x00007f5bac159800 JavaThread "nioEventLoopGroup-3-8" [_thread_in_native, id=17926, stack(0x00007f5b82af6000,0x00007f5b82bf7000)]
  0x00007f5c042ad800 JavaThread "Okio Watchdog" daemon [_thread_blocked, id=17892, stack(0x00007f5b82cf8000,0x00007f5b82df9000)]
  0x00007f5c04134000 JavaThread "OkHttp ConnectionPool" daemon [_thread_blocked, id=17887, stack(0x00007f5b8c2e5000,0x00007f5b8c3e6000)]
  0x00007f5bac158000 JavaThread "nioEventLoopGroup-3-7" [_thread_in_native, id=17881, stack(0x00007f5b8c8eb000,0x00007f5b8c9ec000)]
  0x00007f5bac156000 JavaThread "nioEventLoopGroup-3-6" [_thread_in_native, id=17880, stack(0x00007f5b8cbec000,0x00007f5b8cced000)]
  0x00007f5bac154800 JavaThread "nioEventLoopGroup-3-5" [_thread_in_native, id=17878, stack(0x00007f5b8cced000,0x00007f5b8cdee000)]
  0x00007f5bac152800 JavaThread "nioEventLoopGroup-3-4" [_thread_in_native, id=17877, stack(0x00007f5b8cdee000,0x00007f5b8ceef000)]
  0x00007f5bac151000 JavaThread "nioEventLoopGroup-3-3" [_thread_in_native, id=17876, stack(0x00007f5b8ceef000,0x00007f5b8cff0000)]
  0x00007f5bd03fd000 JavaThread "pool-5-thread-1" [_thread_blocked, id=17875, stack(0x00007f5b8cff0000,0x00007f5b8d0f1000)]
  0x00007f5bac14f800 JavaThread "nioEventLoopGroup-3-2" [_thread_in_native, id=17873, stack(0x00007f5b8e0f2000,0x00007f5b8e1f3000)]
  0x00007f5ba8560800 JavaThread "threadDeathWatcher-4-1" daemon [_thread_blocked, id=17847, stack(0x00007f5bbcbbc000,0x00007f5bbccbd000)]
  0x00007f5c2c008800 JavaThread "DestroyJavaVM" [_thread_blocked, id=17782, stack(0x00007f5c33d4e000,0x00007f5c33e4f000)]
  0x00007f5c2ddd4000 JavaThread "ScriptRunner" [_thread_in_native, id=17845, stack(0x00007f5bbcdbe000,0x00007f5bbcebf000)]
  0x00007f5ba84a2000 JavaThread "pool-1-thread-1" [_thread_blocked, id=17842, stack(0x00007f5bbd0c1000,0x00007f5bbd1c2000)]
  0x00007f5bac141800 JavaThread "nioEventLoopGroup-3-1" [_thread_in_native, id=17832, stack(0x00007f5bbdccb000,0x00007f5bbddcc000)]
  0x00007f5bc5846800 JavaThread "nioEventLoopGroup-2-1" [_thread_in_native, id=17831, stack(0x00007f5bbdfcc000,0x00007f5bbe0cd000)]

Answer 1

Crash logs are not complete, This issue observed on Linux kernel version 2.6.+ . Found to be linux kernel related issue, Poonam has analysed this issue and provided her justification in the blog https://blogs.oracle.com/poonam/entry/analysis_of_strange_hotspot_crashes

Kindly upgrade your kernel version. This issue is due to the xmm registered getting corrupted at runtime on particular linux version that has been fixed in later versions.

Issue is caused by this changeset - https://github.com/torvalds/linux/commit/aa283f49276e7d840a40fb01eee6de97eaa7e012

Please read more on this - https://bugs.openjdk.java.net/browse/JDK-8161645

Answer 2

This is a known issue in JDK 8u45 and earlier.

This bug is referenced as JDK-8067648 (not public), JDK-8049855 and CVE-2015-2659.

It has been fixed in JDK 8u51. So all you need is to install up-to-date JDK version.