Intermittent POST request timeouts observed with ClosedChannelException at processRequest() when content-type is application/x-www-form-urlencoded

Question

Intermittent POST request timeouts observed with ClosedChannelException at processRequest() in FrameworkServlet

Request drops are only seen with POST request when MIME type / content-type is application/x-www-form-urlencoded.

Not seen when MIME type / content-type is changed to application/json for the same service endpoint.

Software under test:

• JDK - 17.0.4
• Spring Boot - 3.1.5
• undertow core, undertow servlet - 2.3.10.Final
• timeout at ingress gateway - 6s

Distribution pattern of successful requests versus 408 timeouts in terms of TPS , when load is run over time.

Rate of successful requests:

Rate of successful requests

Rate of requests timing out:

Rate of requests timing out

Tried parsing the urlencoded body as @ModelAttribute pojo and as @RequestParams map. Issue persisted in both the cases.

Any help would be appreciated.

Thanks :)

Detailed logs:

{"instant":
{"epochSecond":1708007487,"nanoOfSecond":834391795},
"thread":"XNIO-1 task-3","level":"DEBUG","loggerName":"org.springframework.web.method.HandlerMethod","message":"Could not resolve parameter [0] in      public org.springframework.http.ResponseEntity<java.lang.Object> method(Request class ,org.springframework.validation.BindingResult): java.nio.channels.ClosedChannelException","contextMap":{"hostname":"some-microservice","nrfTxId":"nrf-tx-1438526125","","subsystem":"accessToken","xRequestId":"null"},"endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":5665     ,"threadPriority":5,"messageTimestamp":"2024-02-15T14:31:27.834+0000","configuredLevel":"WARN","subsystem":"some microservice","processId":"1","nrfTxId":"nrf-tx-1438526125",","xRequestId":""}

{"instant":
{"epochSecond":1708701707,"nanoOfSecond":439544745},
"thread":"XNIO-1 task-4","level":"ERROR","loggerName":"io.undertow.request","message":"UT005023: Exception handling request to /****/t***","thrown":{"message":"Request processing failed: java.lang.RuntimeException: java.nio.channels.ClosedChannelException","name":"jakarta.servlet.ServletException","extendedStackTrace":[
{"class":"org.springframework.web.servlet.FrameworkServlet","method":"processRequest","file":"FrameworkServlet.java","line":1019},
{"class":"org.springframework.web.servlet.FrameworkServlet","method":"doPost","file":"FrameworkServlet.java","line":914},{"class":"jakarta.servlet.http.HttpServlet","method":"service","file":"HttpServlet.java","line":547},{"class":"org.springframework.web.servlet.FrameworkServlet","method":"service","file":"FrameworkServlet.java","line":885},{"class":"jakarta.servlet.http.HttpServlet","method":"service","file":"HttpServlet.java","line":614},{"class":"io.undertow.servlet.handlers.ServletHandler","method":"handleRequest","file":"ServletHandler.java","line":74},{"class":"io.undertow.servlet.handlers.FilterHandler$FilterChainImpl","method":"doFilter","file":"FilterHandler.java","line":129},{"class":"org.springframework.web.filter.RequestContextFilter","method":"doFilterInternal","file":"RequestContextFilter.java","line":100},{"class":"org.springframework.web.filter.OncePerRequestFilter","method":"doFilter","file":"OncePerRequestFilter.java","line":116},{"class":"io.undertow.servlet.core.ManagedFilter","method":"doFilter","file":"ManagedFilter.java","line":67},{"class":"io.undertow.servlet.handlers.FilterHandler$FilterChainImpl","method":"doFilter","file":"FilterHandler.java","line":131},{"class":"org.springframework.web.filter.FormContentFilter","method":"doFilterInternal","file":"FormContentFilter.java","line":93},{"class":"org.springframework.web.filter.OncePerRequestFilter","method":"doFilter","file":"OncePerRequestFilter.java","line":116},{"class":"io.undertow.servlet.core.ManagedFilter","method":"doFilter","file":"ManagedFilter.java","line":67},{"class":"io.undertow.servlet.handlers.FilterHandler$FilterChainImpl","method":"doFilter","file":"FilterHandler.java","line":131},{"class":"org.springframework.web.filter.ServerHttpObservationFilter","method":"doFilterInternal","file":"ServerHttpObservationFilter.java","line":109},{"class":"org.springframework.web.filter.OncePerRequestFilter","method":"doFilter","file":"OncePerRequestFilter.java","line":116},{"class":"io.undertow.servlet.core.ManagedFilter","method":"doFilter","file":"ManagedFilter.java","line":67},{"class":"io.undertow.servlet.handlers.FilterHandler$FilterChainImpl","method":"doFilter","file":"FilterHandler.java","line":131},{"class":"org.springframework.web.filter.CharacterEncodingFilter","method":"doFilterInternal","file":"CharacterEncodingFilter.java","line":201},{"class":"org.springframework.web.filter.OncePerRequestFilter","method":"doFilter","file":"OncePerRequestFilter.java","line":116},{"class":"io.undertow.servlet.core.ManagedFilter","method":"doFilter","file":"ManagedFilter.java","line":67},{"class":"io.undertow.servlet.handlers.FilterHandler$FilterChainImpl","method":"doFilter","file":"FilterHandler.java","line":131},{"class":"io.undertow.servlet.handlers.FilterHandler","method":"handleRequest","file":"FilterHandler.java","line":84},{"class":"io.undertow.servlet.handlers.security.ServletSecurityRoleHandler","method":"handleRequest","file":"ServletSecurityRoleHandler.java","line":62},{"class":"io.undertow.servlet.handlers.ServletChain$1","method":"handleRequest","file":"ServletChain.java","line":68},{"class":"io.undertow.servlet.handlers.ServletDispatchingHandler","method":"handleRequest","file":"ServletDispatchingHandler.java","line":36},{"class":"io.undertow.servlet.handlers.RedirectDirHandler","method":"handleRequest","file":"RedirectDirHandler.java","line":68},{"class":"io.undertow.servlet.handlers.security.SSLInformationAssociationHandler","method":"handleRequest","file":"SSLInformationAssociationHandler.java","line":117},{"class":"io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler","method":"handleRequest","file":"ServletAuthenticationCallHandler.java","line":57},{"class":"io.undertow.server.handlers.PredicateHandler","method":"handleRequest","file":"PredicateHandler.java","line":43},{"class":"io.undertow.security.handlers.AbstractConfidentialityHandler","method":"handleRequest","file":"AbstractConfidentialityHandler.java","line":46},{"class":"io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler","method":"handleRequest","file":"ServletConfidentialityConstraintHandler.java","line":64},{"class":"io.undertow.security.handlers.AuthenticationMechanismsHandler","method":"handleRequest","file":"AuthenticationMechanismsHandler.java","line":60},{"class":"io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler","method":"handleRequest","file":"CachedAuthenticatedSessionHandler.java","line":77},{"class":"io.undertow.security.handlers.AbstractSecurityContextAssociationHandler","method":"handleRequest","file":"AbstractSecurityContextAssociationHandler.java","line":43},{"class":"io.undertow.server.handlers.PredicateHandler","method":"handleRequest","file":"PredicateHandler.java","line":43},{"class":"io.undertow.servlet.handlers.SendErrorPageHandler","method":"handleRequest","file":"SendErrorPageHandler.java","line":52},{"class":"io.undertow.server.handlers.PredicateHandler","method":"handleRequest","file":"PredicateHandler.java","line":43},{"class":"io.undertow.servlet.handlers.ServletInitialHandler","method":"handleFirstRequest","file":"ServletInitialHandler.java","line":276},{"class":"io.undertow.servlet.handlers.ServletInitialHandler$2","method":"call","file":"ServletInitialHandler.java","line":135},{"class":"io.undertow.servlet.handlers.ServletInitialHandler$2","method":"call","file":"ServletInitialHandler.java","line":132},{"class":"io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1","method":"call","file":"ServletRequestContextThreadSetupAction.java","line":48},{"class":"io.undertow.servlet.core.ContextClassLoaderSetupAction$1","method":"call","file":"ContextClassLoaderSetupAction.java","line":43},{"class":"io.undertow.servlet.handlers.ServletInitialHandler","method":"dispatchRequest","file":"ServletInitialHandler.java","line":256},{"class":"io.undertow.servlet.handlers.ServletInitialHandler$1","method":"handleRequest","file":"ServletInitialHandler.java","line":101},{"class":"io.undertow.server.Connectors","method":"executeRootHandler","file":"Connectors.java","line":393},{"class":"io.undertow.server.HttpServerExchange$1","method":"run","file":"HttpServerExchange.java","line":859},{"class":"org.jboss.threads.ContextHandler$1","method":"runWith","file":"ContextHandler.java","line":18},{"class":"org.jboss.threads.EnhancedQueueExecutor$Task","method":"run","file":"EnhancedQueueExecutor.java","line":2513},{"class":"org.jboss.threads.EnhancedQueueExecutor$ThreadBody","method":"run","file":"EnhancedQueueExecutor.java","line":1538},{"class":"org.xnio.XnioWorker$WorkerThreadFactory$1$1","method":"run","file":"XnioWorker.java","line":1282},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":833}]},"contextMap":{"hostname":"ocnrf-nfaccesstoken-554c4fb465-xqrkk","nrfTxId":"nrf-tx-37398083","ocLogId":"1708701701394_8462_ocnrf-ingressgateway-867d46d696-dcxfw:1708701701397_75_ocnrf-nfaccesstoken-554c4fb465-xqrkk","subsystem":"acc****","xRequestId":"null"},"endOfBatch":false,"loggerFqcn":"io.undertow.UndertowLogger_$logger","threadId":75,"threadPriority":5,"messageTimestamp":"2024-02-23T15:21:47.439+0000","configuredLevel":"WARN","subsystem":"accessToken","processId":"1","nrfTxId":"nrf-tx-37398083","ocLogId":"1708701701394_8462_ocnrf-ingr*****-867d46d696-dcxfw:1708701701397_75_ocn******n-554c4fb465-xqrkk","xRequestId":""}

New to undertow and Spring Boot Framework. Expecting assistance to troubleshoot the issue.

Thanks in advance Soumya