CODEWITHSUNDEEP
timeoutconnection-timeoutreactor-nettyhttp-status-code-502tcp-keepalive
Satyam Pareek
Satyam Pareek

Reputation: 3

How to resolve consistently occuring 502 errors in Spring Reactive based microservice using Netty Server

I have Spring Reactive based microservice which is using Netty Server with version 4.1.101.Final.The complete architecture and flow of my microservice is: API Gateway -> Load Balancer -> PCF (Go Router) -> Spring Reactive App. I am consistently getting 502 errors saying Unexpected EOF on target and no other logs are printed apart from this. For investigating the issue, I performed performance tests using different configuration of load balancer and Api gateway, but I suspect Netty is creating the issue. I feel like Netty is sending a FIN to API gateway before 60 seconds and timeout is configured to 60 seconds in Api Gateway.

I was trying to figure out the by default connection timeout of netty and trying to set the connection-timeout to 60 seconds by adding it in application.yml using:

server:
   netty:
     connection-timeout: 60000

I have enabled the logs of reactor nettty and kept the logging level as DEBUG. But I cannot see any logs related to this which can ensure the connection timeout. After I start my server and hit any request...then I see continuos logs saying New http connection, requesting read. Then It establishes a connection and when the request is served it immediately starts requesting for new connection. It does not look like there is any connection timeout or keep-alive timeout being configured.

I also tried to add keep-alive timeout for netty server by creating a Custom Netty Config.

public class NettyCustomConfig implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory>{
@Override
public void customize(NettyReactiveWebServerFactory factory){
   factory.addServerCustomizers(httpServer->
      httpServer.tcpConfiguration(tcp->
         tcp.childOption(ChannelOption.SO_KEEPALIVE,true)
          .childOption(EpollChannelOption.TCP_KEEPINTVL,2)
          .childOption(EpollChannelOption.TCP_KEEPIDLE,2)
          .childOption(EpollChannelOption.TCP_KEEPCNT,2)
));
}
}

After this also, I cannot see any logs which can give any information about connection getting dropped or reused after keep-alive timeout.I also cannot see logs that show notification being sent after connection is idle and connection is dropped before a new one is created.

I want to capture the connection-timeout and keep-alive timeouts for the connections being used for processing the requests and catch the connections which are being dropped on being idle or timeout so that I can solve my 502 errors by configuring correct timeout values.

Upvotes: 0

Views: 166

Answers (0)

Related Questions