Assert System.out/err in testcase

Question

I'm looking at porting an old junit4 extension to junit5 by creating an extension. One feature that I'm thus far failing on is to fail any test method that writes on System.out or System.err, and have the failure point to that specific test.

I tried using TestExecutionListener#reportingEntryPublished together with junit.jupiter.extensions.autodetection.enabled=true, and I get to that method, but it is AFAICT too late to actually change the status of the test to FAILED at this point. Or can I?

I started playing with BeforeAllCallback and AfterAllCallback and do redirects of out/err and assert on a string buffer instead. I could possibly be able to get that to work, but that seems like something that I would be likely to screw up and and up finding bugs years from now :) Our runner for junit4 does something like this, but it is really overcomplicated and I wouldn't wish it on anyone to try to maintain something similar.

Any way to do this nicely in junit5?

Answer 1

What I ended up doing was to create a class implementing BeforeEachCallback, AfterEachCallback and redirecting each of stderr/out during beforeEach to a custom printstream each that stores all the output received. Then during afterEach, I ask the printstream if it has received anything. If it has, then I throw AssertionError that the custom stream has generated based on a modified stacktrace from when the first print-call was made. Then resets out/err to the original streams. This allows for some tricks to allow the output to be printed to the console without throwing an error and still flag the testcase as failed with a custom message about disallowing output.

There are some potential pitfalls in this solution as e.g. an error in one of my other extension classes that tries to call Logger will create some fishy cases. For our use case, it should be sufficient.

Due to my employer having a lot of hoops to jump through to be allowed to share code externally, I'm unfortunately unable to do so for anyone else needing to do the same thing, but should be able to figure out how to do it decently based on the above.