Issue with neo4j driver when terminating java CLI

Question

Recently I had upgrading (org.neo4j.driver) neo4j-java-driver from 1.4.4 to latest 1.7.5 as I wanted to start using temporal types within db. After upgrading all seemed to work smoothly until I've started playing with CLI tools (custom tools prepared in my code), which when are terminating are falling into endless loop:

Warning: unterminated thread Neo4jDriverIO-2-10 [23] [sun.nio.ch.EPollArrayWrapper.epollWait(Native Method), sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269), sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93), sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86), sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97), org.neo4j.driver.internal.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62), org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753), org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30), java.lang.Thread.run(Thread.java:748)]
WARNING: Warning: unterminated thread Neo4jDriverIO-2-10[sun.nio.ch.EPollArrayWrapper.epollWait(Native Method), sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269), sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93), sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86), sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97), org.neo4j.driver.internal.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62), org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753), org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30), java.lang.Thread.run(Thread.java:748)]
Warning: unterminated thread Neo4jDriverIO-2-9 [22] [sun.nio.ch.EPollArrayWrapper.epollWait(Native Method), sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269), sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93), sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86), sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97), org.neo4j.driver.internal.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62), org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753), org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30), java.lang.Thread.run(Thread.java:748)]
WARNING: Warning: unterminated thread Neo4jDriverIO-2-9[sun.nio.ch.EPollArrayWrapper.epollWait(Native Method), sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269), sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93), sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86), sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97), org.neo4j.driver.internal.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62), org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753), org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30), java.lang.Thread.run(Thread.java:748)]
[..]

I've tried to directly call close() method on driver, just before exiting from CLI program. It had helped a bit as the program closed, but I still could see following stacks:

INFO: Closing driver instance 1162128198
Nov 14, 2019 5:03:25 PM org.neo4j.driver.internal.logging.JULogger info
INFO: Closing connection pool towards neo4j:7687
Warning: unterminated thread globalEventExecutor-1-2 [36] [sun.misc.Unsafe.park(Native Method), java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:92), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:245), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30), java.lang.Thread.run(Thread.java:748)]
WARNING: Warning: unterminated thread globalEventExecutor-1-2[sun.misc.Unsafe.park(Native Method), java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:92), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:245), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30), java.lang.Thread.run(Thread.java:748)]
Warning: unterminated thread globalEventExecutor-1-2 [36] [sun.misc.Unsafe.park(Native Method), java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215), java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078), java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:92), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:245), org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30), java.lang.Thread.run(Thread.java:748)]
[..] 

Any ideas why its not closing properly (or why its spawning netty threads?) and why the heck I see such stacks?

Note:

• that ofcourse in case of 1.4.4 no such issues were/are observed.
• I am using direct bolt connection to external single neo4j db instance.

UPDATE

So I've found the reason why I am seeing such logs. Basically Neo4jDriverIO threads are now daemon threads and my CLI code had been checking if there are any threads which .isDaemon is true and was waiting till those gets closed.

In first case they were never closing as .close() method from neo4j driver was supposed to be called by JVM shutdown hook, which was never rised as we were in this isDaemon loop.

Still remains 2nd question, why globalEventExecutor-1-2 are not closed during close() method, but only after some time (which means those are closed in async fashion). It maybe a bug, but will dig a bit more.