Java StackOverflowError at java.io.PrintStream.write(PrintStream.java:480) and no further stack trace

Question

I am running a Java program written by another person on more data than the program was initially designed for, e.g. 10-times longer input files, roughly quadratic runtime. I encountered different problems and now aim to solve them bit by bit.

During execution when a lot of output has already been printed (redirected to a file) I get following output:

Exception in thread "main" java.lang.StackOverflowError  
        at java.io.PrintStream.write(PrintStream.java:480)  
        [...]  
        at java.io.PrintStream.write(PrintStream.java:480)

The stack trace is the first thing that confuses me as it is a long repetition of just the same line again and again. Furthermore, it gives no intention where in the code or the execution the problem occurred.

My thoughts / research

• StackOverflowError
  • may indicate too less memory. With -Xmx110G flag I provided 110 G memory and monitored it while execution, only up to ~32 G were used. So this is probably not the problem here.
  • may be thrown due to programming mistakes causing an infinite loop. However, I cant really check this since I am not familiar enough with the code and the stack trace does not help me finding the location of the problem in the code.
  • [hypothesis] may be caused, because the writing of output is slower than execution and new print/write calls. Still, why is there no further stack trace? How could I check and fix this?

• PrintStream

  • only code fragments after searching for "PrintStream"

    // reset output stream to suppress the annoying output of the Apache batik library. Gets reset after lib call.  
    OutputStream tmp=System.out;  
    System.setOut(new PrintStream(new org.apache.commons.io.output.NullOutputStream()));  
    drawRes.g2d.stream(new FileWriter(svgFilePath), false);      
    System.setOut(new PrintStream(tmp));  
    

  • [hypothesis] the write to void / null does not work
  • [workaround] if skip the change of output stream and just "live" with the big output created the programm seems to run (into other problems, but this is another case). Any ideas why this is happening?

Call for advice
If you have any advice on what is going one, what the Java code specifically does, please help me understand it. Especially the stack trace frustrates me as it provides no place to begin the fixing. I am also thankful for a general approach on how to tackle this problem, get a stack trace, fix the code to avoid StackOverflow, etc.

Some system environment facts

• Linux machine
• 128 G memory

• Java

  openjdk version "1.8.0_121"  
  OpenJDK Runtime Environment (IcedTea 3.3.0) (suse-28.1-x86_64)  
  OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
  

• Please ask if you need more information!

Notes

• I am rather new to Java so I am grateful for every advice (the program is not written by me)
• This is my first post on stackoverflow, please inform me where I can improve my style of asking and formatting
• I am no native English speaker, so please excuse mistakes and feel free to ask for understanding or correct me

Thanks all for your replies!

Answer 1

These two lines look suspicious:

OutputStream tmp=System.out;
//...
System.setOut(new PrintStream(tmp));

System.out is already a PrintStream, so IMHO the lines should read

PrintStream tmp=System.out;
//...
System.setOut(tmp);

What happens otherwise is that you have a near endless wrapping of PrintStreams within PrintStreams. The nesting of PrintStreams is only limited through Java heap space - but the call level nesting is much lower.

---

To verify the hypothesis I created a small test program that first wraps System.out 20 times and prints a stacktrace to verify the call chain. After that it wraps System.out 10_000 times and produces a StackOverflowException.

import java.io.OutputStream;
import java.io.PrintStream;

public class CheckPrintStream {
    public static void main(String[] args) {
        PrintStream originalSystemOut = System.out;
        System.setOut(new PrintStream(System.out) {
            @Override
            public void write(byte buf[], int off, int len) {
                originalSystemOut.write(buf, off, len);
                if (len > 2) {
                    new RuntimeException("Testing PrintStream nesting").printStackTrace(originalSystemOut);
                }
            }
        });

        for (int i = 0; i < 20; i++) {
            wrapSystemOut();
        }
        System.out.println("Hello World!");

        for (int i = 20; i < 10_000; i++) {
            wrapSystemOut();
        }
        System.out.println("crash!");
    }

    private static void wrapSystemOut() {
        OutputStream tmp = System.out;
        System.setOut(new PrintStream(System.out));
    }
}

A nesting of around 6000 to 7000 PrintWriters is sufficient to produce a stack overflow.