How to prevent physical memory consuming when running parallel Java processes

Question

I have big list (up to 500 000) of some functions. My task is to generate some graph for each function (it can be do independently from other functions) and dump output to the file (it can be several files). The process of generating graphs can be time consuming.

I also have server with 40 physical cores and 128GB ram.

I have tried to implement parallel processing using java Threads/ExecutorPool, but it seems not to use processors all resources. On some inputs the program takes up to 25 hours to run and only 10-15 cores are working according to htop.

So the second thing I've tried is to create 40 distinct processes (using Runtime.exec) and split the list among them. This method uses processor all resources (100% load on all 40 cores) and speedups performance up to 5 times on previous example (it takes only 5 hours which is reasonable for my task). But the problem of this method is that, each java process runs separately and consumes memory independently from others. Is some scenarios all 128gb of ram is consumed after 5 minutes of parallel work. One solution that I am using now is to call System.gc() for each process if Runtime.totalMemory > 2GB. This slows down overall performance a bit (8 hours on previous input) but lefts memory usage in reasonable boundaries. But this configuration works only for my server. If you run it on the server with 40 core and 64GB run, you need to tune Runtime.totalMemory > 2GB condition.

So the question is what is the best way to avoid such aggressive memory consuming?

Is it normal practice to run separate processes to do parallel jobs?

Is there any other parallel method in Java (maybe fork/join?) which uses 100% physical resources of processor.

Answer 1

One of the problems here: it is still very hard to understand how many threads, cores, ... are actually available.

My personal suggestion: there are several articles on the java specialist newsletter which do a very deep dive into this subject.

For example this one: http://www.javaspecialists.eu/archive/Issue135.html

or a more recent new, on "the number of available processors": http://www.javaspecialists.eu/archive/Issue220.html

Answer 2

You don't need to call System.gc() explicitly! The JVM will do it automatically when needed, and almost always does it better. You should, however, set the max heap size (-Xmx) to a number that works well.

If your program won't scale further you have some kind of congestion. You can either analyse your program and your java- and system settings and figure out why, or run it as multiple processes. If each process is multi-threaded, then you may get better performance using 5-10 processes instead of 40.

Note that you may get higher performance with more than one thread per core. Fiddle around with 1-8 threads per core and see if throughput increases.

From your description it sounds like you have 500,000 completely independent items of work and that each work item doesn't really need a lot of memory. If that is true, then memory consumption isn't really an issue. As long as each process has enough memory so it doesn't have to gc very often then gc isn't going to affect the total execution time by much. Just make sure you don't have any dangling references to objects you no longer need.