CODEWITHSUNDEEP
javaspring-bootdockerkubernetes
Pankaj Garg
Pankaj Garg

Reputation: 1

Requests get dropped during springboot service pod rollout

I am using the Kubernetes cluster with docker. When I deploy the java services[springboot] some requests get dropped(for a couple of secs) with the following error.

exception=org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'controller': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!), stackTrace=[org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton() at line: 208]

I am already using livenessProbe & readinessProbe. Java Version: 12 SpringBoot Version: 2.1.5.RELEASE Hibernate Version: 2.4.3 with Postgres DB

As per my knowledge, it is happening due to the closing of the application context while executing some requests. Ideally, it should not be.

can anyone help here ?

Upvotes: 0

Views: 946

Answers (1)

Chris
Chris

Reputation: 200

the problem is not actually springboot, but rather the way Kubernetes stops pods. at the moment when a pod from your old deployment/replicaset is being terminated (or rather actually set to state "terminating"), 2 things happen simultaneously:

A) pod is removed from service endpoints, so it does no longer receive new requests

B) pod container gets a SIGTERM, so apps can gracefully shutdown

so what you are seeing here is basically active requests that are being processed when the context gets shut down (as you already found out)

there are (at least) two solutions:

1 in kubernetes pod definition:

Kubernetes pods can be configured with a pre-stop hook that get executes a command in between A and B. depending on your app, a simple "sleep" for a couple (milli)seconds should be sufficient, leaving the app enough time to finish the current requests before shutting down. theres nice docu from google that goes more into detail: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace

2 in SpringBoot:

you can make Java wait for finishing up running tasks when receiving the shutdown interrupt. this is (imho) nicely explained here: https://www.baeldung.com/spring-boot-graceful-shutdown

Beware: kubernetes default graceful shutdown timeout is 30seconds, then the pod is forcefully removed. but as usual you can configure this timeout in terminationGracePeriodSeconds (also described in the google blog in (1)

Upvotes: 1

Related Questions