CODEWITHSUNDEEP
javaspringspring-boot
elyograg
elyograg

Reputation: 789

Spring Boot -- delay boot's shutdown hook until my shutdown hook is finished

I have a spring boot application running on two servers behind a load balancer.

I've defined a shutdown hook that sets the /healthcheck endpoint in my application to return a 404 error, waits 30 seconds for everything to finish up, and then halts the JVM. The load balancer is configured to put the server into maintenance mode if it see a 404 error. (This is different than a "DOWN" status -- in-flight connections are allowed to finish in maintenance mode)

The problem is that the load balancer never sees those 404 responses. Instead, it gets a layer 4 TCP connection refused, putting the server in DOWN status on the load balancer. I think this is happening because one of the other shutdown hooks (probably Spring, Boot, or Tomcat) is removing the TCP binding.

Is it possible to ensure that my shutdown hook runs to completion before any other shutdown hooks are called, or am I going to need to alter my init script to set the 404 status on healthcheck and wait the thirty seconds there before it stops the process?

Upvotes: 1

Views: 1071

Answers (2)

elyograg
elyograg

Reputation: 789

I never did figure out how to handle this with a shutdown hook.

What I did to solve the problem is add an endpoint with the URL path "/lbdisable" which causes the "/healthcheck" endpoint to return a 404 error.

In the init script, I call "http://127.0.0.1:PORT/lbdisable" with curl, wait for 40 seconds, and then kill the program. The switch from 200 to 404 on the health check causes the load balancer to put the server into maintenance mode. That mode allows in-flight requests to finish, while ensuring that no new requests are sent to the server.

With these changes, I can now restart the program on both servers, with a two minute delay between the restarts. Clients have no knowledge that anything has gone down, and there is zero loss.

Upvotes: 1

bichito
bichito

Reputation: 1426

Shutdown hooks are threads. I can not see how you could enforce any execution order

Upvotes: 0

Related Questions