Evaluation - OutOfMemoryError: unable to create new native thread

Question

Though this is a typical error messages reported by various Stackoverflow users, my question is related to how to evaluate whether the proposed solutions solve the problem.

I read various discussions & articles related to this error and most solutions drill-down to Linux ulimits and I guess that seems to be the case for me too.

My ulimit values are:

STACK 10240k, CORE 0k, NPROC 1024, NOFILE 4096;

I guess the issue could be with NOPROC / NOFILE being too low (with just the default values).

However, I wanted to know if there is an exact way to identify the root cause say the NOPROC has been exceeded etc., and if there is a way to evaluate exactly how many processes / file handles are currently being used; Or are there some other issues that I should focus on which can be statistically evaluated?

FYI, when this issue occurred, heapdump wasn't enabled and there is no threadump data at the point of error.

Appreciate your inputs towards evaluating and fixing this.

Here is the brief stacktrace:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)

Here are the System values:

OS:Red Hat Enterprise Linux Server release 6.3 (Santiago)
uname:Linux 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE 0k, NPROC 1024, NOFILE 4096, AS infinity
load average:0.11 0.10 0.03
CPU:total 32 (8 cores per cpu, 2 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, ht, tsc, tscinvbit, tscinv

/proc/meminfo:
MemTotal:       74206252 kB
MemFree:         2788244 kB
Buffers:         1042212 kB
Cached:         58454988 kB
SwapCached:         2860 kB
Active:         38242540 kB
Inactive:       29129604 kB

Here is the info from the JVM Crash report - hs_err_pidxxxxx.log:

# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
...
#  Out of Memory Error (gcTaskThread.cpp:46), pid=20396, tid=140365307795200

# JRE version:  (7.0_80-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

Current thread (0x00007fa95400a800):  JavaThread "Unknown thread" [_thread_in_vm, id=20458, stack(0x00007fa9583f5000,0x00007fa9584f6000)]
Stack: [0x00007fa9583f5000,0x00007fa9584f6000],  sp=0x00007fa9584f4540,  free space=1021k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x9a320a]  VMError::report_and_die()+0x2ea
V  [libjvm.so+0x498d3b]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V  [libjvm.so+0x55943a]  GCTaskThread::GCTaskThread(GCTaskManager*, unsigned int, unsigned int)+0x11a
V  [libjvm.so+0x5589b8]  GCTaskManager::initialize()+0x2b8
V  [libjvm.so+0x843438]  ParallelScavengeHeap::initialize()+0x6f8
V  [libjvm.so+0x97509a]  Universe::initialize_heap()+0xca
V  [libjvm.so+0x976269]  universe_init()+0x79
V  [libjvm.so+0x5b2f25]  init_globals()+0x65
V  [libjvm.so+0x95db4d]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x1ed
V  [libjvm.so+0x63b2e4]  JNI_CreateJavaVM+0x74
C  [libjli.so+0x2f8e]  JavaMain+0x9e
Java Threads: ( => current thread )
Other Threads:
=>0x00007fa95400a800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=20458, stack(0x00007fa9583f5000,0x00007fa9584f6000)]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events

Answer 1

I wanted to know if there is an exact way to identify the root cause say the NOPROC has been exceeded etc

The JVM, like any other software, ultimately has to talk to the kernel through syscalls. To spawn new threads it has to use the clone syscall which can return various error codes (documented in the man pages). You can use strace to log the syscalls and see their error codes, which can provide more fine-grained information than the OOME.