Spring Boot scheduled Runnable tasks continue executing even after server is shut down with an actuator

Question

I'm currently developing a Spring-based web platform which makes use of several scheduled processes that access a central database. I wanted to introduce actuators for shutdown and restart. However, I am experiencing an issue: even though the shutdown request is accepted with a 200 response, the application context begins shutting down:

2022-10-10 17:37:25.235  INFO 9067 --- [       Thread-3] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2022-10-10 17:37:25.240  INFO 9067 --- [       Thread-3] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.

And after that, I repeatedly get the following exception:

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is java.lang.IllegalStateException: EntityManagerFactory is closed
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467) ~[spring-orm-5.3.16.jar:5.3.16]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.16.jar:5.3.16]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.16.jar:5.3.16]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) ~[spring-aop-5.3.16.jar:5.3.16]
    at usi.si.seart.service.TaskService$TaskServiceImpl$$EnhancerBySpringCGLIB$$4bf83a1d.getNext(<generated>) ~[classes/:na]
    at usi.si.seart.scheduling.TaskRunner.run(TaskRunner.java:68) ~[classes/:na]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.16.jar:5.3.16]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.lang.IllegalStateException: EntityManagerFactory is closed
    at org.hibernate.internal.SessionFactoryImpl.validateNotClosed(SessionFactoryImpl.java:547) ~[hibernate-core-5.6.5.Final.jar:5.6.5.Final]
    at org.hibernate.internal.SessionFactoryImpl.createEntityManager(SessionFactoryImpl.java:636) ~[hibernate-core-5.6.5.Final.jar:5.6.5.Final]
    at org.hibernate.internal.SessionFactoryImpl.createEntityManager(SessionFactoryImpl.java:158) ~[hibernate-core-5.6.5.Final.jar:5.6.5.Final]
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.createNativeEntityManager(AbstractEntityManagerFactoryBean.java:585) ~[spring-orm-5.3.16.jar:5.3.16]
    at jdk.internal.reflect.GeneratedMethodAccessor112.invoke(Unknown Source) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.invokeProxyMethod(AbstractEntityManagerFactoryBean.java:487) ~[spring-orm-5.3.16.jar:5.3.16]
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean$ManagedEntityManagerFactoryInvocationHandler.invoke(AbstractEntityManagerFactoryBean.java:734) ~[spring-orm-5.3.16.jar:5.3.16]
    at com.sun.proxy.$Proxy175.createNativeEntityManager(Unknown Source) ~[na:na]
    at org.springframework.orm.jpa.JpaTransactionManager.createEntityManagerForTransaction(JpaTransactionManager.java:485) ~[spring-orm-5.3.16.jar:5.3.16]
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:410) ~[spring-orm-5.3.16.jar:5.3.16]
    ... 17 common frames omitted

So in spite of the fact that the server is reported as "shutting down", the scheduled processes still continue executing. My scheduler definition within the configuration class is as follows:

@Bean
public ThreadPoolTaskScheduler taskScheduler() {
    ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
    threadPoolTaskScheduler.setClock(Clock.systemUTC());
    threadPoolTaskScheduler.setPoolSize(3);
    threadPoolTaskScheduler.setThreadNamePrefix("PlatformScheduler");
    threadPoolTaskScheduler.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
    threadPoolTaskScheduler.setRemoveOnCancelPolicy(true);
    threadPoolTaskScheduler.initialize();

    threadPoolTaskScheduler.schedule(getRepoMaintainer(), new CronTrigger("0 0 0 * * SUN"));
    threadPoolTaskScheduler.schedule(getTaskCleaner(), new CronTrigger("0 */15 * * * *"));
    threadPoolTaskScheduler.scheduleWithFixedDelay(getTaskRunner(), 500);

    return threadPoolTaskScheduler;
}

Where the getRepoMaintainer, getTaskCleaner and getTaskRunner are all methods that produce custom Runnable instances: RepoMaintainer, TaskCleaner and TaskRunner respectively.

Answer 1

Unfortunately, none of the fixes suggested by the other users worked. In the end, I managed to solve my problem by virtue of a custom ThreadPoolTaskScheduler implementation:

ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler() {
    private final Set<ScheduledFuture<?>> futures = new HashSet<>();

    @Override
    public ScheduledFuture<?> scheduleWithFixedDelay(Runnable task, long delay) {
        ScheduledFuture<?> future = super.scheduleWithFixedDelay(task, delay);
        futures.add(future);
        return future;
    }

    @Override
    public ScheduledFuture<?> schedule(Runnable task, Trigger trigger) {
        ScheduledFuture<?> future =  super.schedule(task, trigger);
        futures.add(future);
        return future;
    }

    @Override
    public void shutdown() {
        futures.forEach(future -> future.cancel(true));
        super.shutdown();
    }
};

This way I ensure that all scheduled tasks are cancelled pre-shutdown no matter what.

Answer 2

As you have specified that your custom destroyMethod is "shutdown" have you defined that method with the required destruction statements?

Something like this,

public void shutdown() {
    threadPoolTaskScheduler.shutdown();
}

Edit- Most probably the error here is, even though the shutdown has been initiated the tasks are still running but the container is shutting down. This releases the resources required for the task to execute and the error is thrown. What you can do here is perform a graceful shutdown where you prevent the shutdown of the container until the tasks have finished executing.

Your issue is similar to this one and you can try out his solution.

Or, you can configure a termination period like so, setAwaitTerminationSeconds(*seconds*) Check more here. But then you might face this issue.