Response Body Bad chunk header - I/O error on GET request

Question

Setup - 2 Rest services, spring-boot 2.x, embedded tomcat 8.x

Service A calls Service B using restTemplate, in a test loop running for 300 times, on every 100th interval within the loop throws an exception "Bad chunk header exception".

Enabled debugging (<logger name="org.apache.http" level="DEBUG"/>), following is the analysis:

First 99 iterations - Request/Response headers have Connection: keep-alive and body/stream ends with carriage return \r\n

Request Headers

"GET /rest/customers/1000000030 HTTP/1.1[\r][\n]"
"Accept-Language: en-us[\r][\n]"
"Accept: application/json[\r][\n]"
"Content-Type: application/json[\r][\n]"
"Host: localhost:9192[\r][\n]"
"Connection: Keep-Alive[\r][\n]"
"User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_25)[\r][\n]"
"Accept-Encoding: gzip,deflate[\r][\n]"
"[\r][\n]"

Response Headers

"HTTP/1.1 200 [\r][\n]"
"X-Content-Type-Options: nosniff[\r][\n]"
"X-XSS-Protection: 1; mode=block[\r][\n]"
"Cache-Control: no-cache, no-store, max-age=0, must-revalidate[\r][\n]"
"Pragma: no-cache[\r][\n]"
"Expires: 0[\r][\n]"
"X-Frame-Options: DENY[\r][\n]"
"X-Content-Type-Options: nosniff[\r][\n]"
"X-Content-Type-Options: nosniff[\r][\n]"
"X-Content-Type-Options: nosniff[\r][\n]"
"Cache-Control: no-cache, no-store, max-age=0, must-revalidate[\r][\n]"
"Pragma: no-cache[\r][\n]"
"Expires: 0[\r][\n]"
"Connection: close[\r][\n]"
"Transfer-Encoding: chunked[\r][\n]"
"Date: Wed, 03 Oct 2018 00:31:53 GMT[\r][\n]"
"Content-Type: application/json;charset=UTF-8[\r][\n]"
"[\r][\n]"
"{"customerNumber":"1000000030"}[\r][\n]"

On 100th call - Request headers have Connection: keep-alive, but response haader has Connection: close and body/stream WITHOUT carriage return \r\n
http-outgoing-0 << "end of stream"
I/O error on GET request for "http://localhost:9192/rest/customers/1000000030": Bad chunk header

Request Headers

"GET /rest/customers/1000000030 HTTP/1.1[\r][\n]"
"Accept-Language: en-us[\r][\n]"
"Accept: application/json[\r][\n]"
"Content-Type: application/json[\r][\n]"
"Host: localhost:9192[\r][\n]"
"Connection: Keep-Alive[\r][\n]"
"User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_25)[\r][\n]"
"Accept-Encoding: gzip,deflate[\r][\n]"
"[\r][\n]"

Response Headers

"HTTP/1.1 200 [\r][\n]"
"X-Content-Type-Options: nosniff[\r][\n]"
"X-XSS-Protection: 1; mode=block[\r][\n]"
"Cache-Control: no-cache, no-store, max-age=0, must-revalidate[\r][\n]"
"Pragma: no-cache[\r][\n]"
"Expires: 0[\r][\n]"
"X-Frame-Options: DENY[\r][\n]"
"X-Content-Type-Options: nosniff[\r][\n]"
"X-Content-Type-Options: nosniff[\r][\n]"
"X-Content-Type-Options: nosniff[\r][\n]"
"Cache-Control: no-cache, no-store, max-age=0, must-revalidate[\r][\n]"
"Pragma: no-cache[\r][\n]"
"Expires: 0[\r][\n]"
"Connection: close[\r][\n]"
"Transfer-Encoding: chunked[\r][\n]"
"Date: Wed, 03 Oct 2018 00:31:53 GMT[\r][\n]"
"Content-Type: application/json;charset=UTF-8[\r][\n]"
"[\r][\n]"
"{"customerNumber":"1000000030"}" <- MISSING \r\n

a. Why is the server closing the connection after 100 calls?

Seems like this is the cause- Tomcat 8 with NIO connectors defaults:

• Maximum connections that can be handled by each server are maxConnections = 10000
• Each connection timeout connectionTimeout = 60 seconds
• Maximum threads defines within each connection how many simultaneous requests can be handled maxThreads = 200
• Minimum spare threads that are always available (not connections) = minSpareThreads = 10
• Maximum number keep alive requests within a connection maxKeepAliveRequests = 100 (this seems to be closing the connection, and does not send carriage return as part of response body generating Buffer Read I/O exception)

b. When closing connection why server is not sending the termination or carriage return? Is this Spring or tomact issue?

c. How or what setting need to be changed to avoid this? maxKeepAliveRequests is not exposed in Spring application properties, only way to override is to implement a customer container implementing WebServerFactoryCustomizer and then manually overriding.

Please advise if this a bug on connection close event on the server?

Answer 1

I met this issue when using RestTemplate exchange function, I just filter the Connection header, then the error disappears.