CODEWITHSUNDEEP
javamultithreadingmemory
Colin
Colin

Reputation: 10820

Java out of memory error when creating thread

I have found a lot of questions here about the error I'm getting, but none of the solutions are working for me. I hope this isn't counted as a duplicate question. I'm using a data analysis software tool written in Java, and I'm getting the following error:

[286.098s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. 
Exception in thread "main" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
            at java.base/java.lang.Thread.start0(Native Method)
            at java.base/java.lang.Thread.start(Thread.java:803)
            at edu.mit.compbio.ChromHMM.ChromHMM.trainParametersParallel(ChromHMM.java:9729)
            at edu.mit.compbio.ChromHMM.ChromHMM.buildModel(ChromHMM.java:901)
            at edu.mit.compbio.ChromHMM.ChromHMM.main(ChromHMM.java:12014)

The only argument I give to the java interpreter is -Xmx10000M

Most of the people I've seen with this issue are trying to create a huge number of threads, and that's the source of their problem. But here the program is only trying to create a single thread. I am running it on a computing cluster and have requested 128GB of memory on the node. As I understand it, threads are created in memory outside the java heap, so there should still be 118GB available. I've tried both increasing the decreasing the 10000M size that I give to java, but it doesn't make a difference. I also added -Xss1000M to give a larger stack size to the thread, but again got the same error.

I successfully run this tool with a smaller amount of input data, and have only started to experience this problem with larger datasets, I just can't figure out what knobs to turn so that there is enough memory available. I've watched the process in top and it never uses more than 10% of available memory, so it seems like there's some issue with how that memory is distributed within Java. Is there anything else I can try, or at this point is it probably just that the tool needs to be coded differently to be able to handle larger amounts of data?

Upvotes: 1

Views: 2756

Answers (1)

Jose1755
Jose1755

Reputation: 478

I had the same problem in a centOS/Red Hat machine. You are reaching the threads limit, for the user, process, or an overall limit

In my case there was a limit on the number of threads a user can have. Which can be checked with, the line saying max user processes

ulimit -a

You can see how many threads are running using this command

$ ps -elfT | wc -l

To get how many threads your process is running (you can get your process pid using top or ps aux):

$ ps -p <PROCESS_PID> -lfT | wc -l

The /proc/sys/kernel/threads-max file provides a system-wide limit for the number of threads. The root user can change that value

To change the limits (in this case to 4096 threads):

$ ulimit -u 4096

You can find more info here for Red Hat/centOs http://www.mastertheboss.com/jboss-server/jboss-monitoring/how-to-solve-javalangoutofmemoryerror-unable-to-create-new-native-thread

Upvotes: 2

Related Questions