Undertow HTTP2: request threads exhausted with threads stuck in org.xnio.conduits...awaitReadable/Writable operations

Question

Something occasionally happened for one random instance of 5 available instances few times in a day. Worker threads get stucked, after exhausting all the worker threads service fail to respond a health check and get restarted. We observe that problem occurs with POST/PUT/PATCH requests. Those stucked requests runs for tens of minutes and then are reported as failed with status code 400 || 500. Blocked threads grows quickly and in few minutes could exhauste all pool, and they grow only on a single instance other 4 works fine.

Spring boot version 3.1.4 / 3.0.11 Spring Cloud Gateway.(Netty) <->(http2 SSL) Spring based service with REST endpoint(Undertow).

We add an Undertow StuckThreadDetector and it reports a stacktrace like:

java.lang.Throwable: null
    at java.base@17.0.2/java.lang.Object.wait(Native Method)
    at java.base@17.0.2/java.lang.Object.wait(Object.java:338)
    at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.awaitReadable(AbstractFramedStreamSourceChannel.java:354)
    at org.xnio.conduits.StreamSourceChannelWrappingConduit.awaitReadable(StreamSourceChannelWrappingConduit.java:71)
    at org.xnio.conduits.ConduitStreamSourceChannel.awaitReadable(ConduitStreamSourceChannel.java:151)
    at io.undertow.channels.DetachableStreamSourceChannel.awaitReadable(DetachableStreamSourceChannel.java:77)
    at io.undertow.server.HttpServerExchange$ReadDispatchChannel.awaitReadable(HttpServerExchange.java:2305)
    at org.xnio.channels.Channels.readBlocking(Channels.java:345)
    at io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:201)
    at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:176)
    at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:162)
    at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:153)
    at java.base@17.0.2/java.io.FilterInputStream.read(FilterInputStream.java:82)
    at java.base@17.0.2/java.io.PushbackInputStream.read(PushbackInputStream.java:135)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver$EmptyBodyCheckingHttpInputMessage.<init>(AbstractMessageConverterMethodArgumentResolver.java:331)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:172)
    at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:163)
    at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:136)
    at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:122)
    at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:179)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:146)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)

java.lang.Throwable: null
    at java.lang.Object.wait(Object.java)
    at io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.awaitWritable(AbstractFramedStreamSinkChannel.java:329)
    at io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.awaitWritable(AbstractFramedStreamSinkChannel.java:313)
    at org.xnio.conduits.StreamSinkChannelWrappingConduit.awaitWritable(StreamSinkChannelWrappingConduit.java:99)
    at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
    at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
    at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:2141)
    // ... 183 frames truncated

This guys mention similar issue https://access.redhat.com/solutions/6988036. - but there are no info how they fix it.

We add an Undertow StuckThreadDetector to catch the thread's stacktrace. We are going to try to BlockingReadTimeoutHandler BlockingWriteTimeoutHandler to fail fast when there are a problems in undertow awaitReadable/awaitWritable, but we are still looking for a root cause of such a random issue.

Answer 1

Some details are present in the https://issues.redhat.com/browse/UNDERTOW-1639 which to me kind of acknowledges that the awaitReadable can still block in certain conditions. A better solution is to use the non-blocking awaitReadable using the BlockingReadTimeoutHandler.

Quoting from the bug comment by the developer:

I've discovered that this is still possible to hit with my fix, but substantially less likely.
I think the check prior to lock.wait in awaitReadable should be updated to check 'isOpen', otherwise invocations after a close may wait forever.

Answer 2

Ohhh... this tricky issue with Underthow... When I used Tomcat as an application server - there were no problems, however there was a necessity to change it to Underthow...

Hope, you will find a solution!