Correctly shutting down a work stealing thread pool

Question

Given a work stealing thread pool system where each work item can generate new tasks in a threads local work queue - that can spill out to a global queue if full.

How would you safely and efficiently coordinate the shutdown of such a system? Assuming you have basic atomic operations and critical section locks available only.

To clarify some more and simplify. Say each thread grabs tasks from it's local work queue only (no stealing between other threads queues to simplify). If it's local work queue is exhausted it will take a lock on the global work queue and steal work to add to its local work queue. The local work queues require no locks as they are specific to each worker thread.

Using a simple flag or atomic count of 'active' worker threads won't work due to cases where other workers may spill new work onto the global queue where from another worker threads view it may have thought it was the only worker left with work.

All workers should exit only when there is no work left.

Answer 1

The biggest requirement would be to have some way of saving the definition of each task so the state of pending tasks can be saved to persistent storage. Then implement a "stop" flag (with a mutex on it). The method to get a task from the pool for execution checks that flag and, if it's set, returns a "terminate work thread" indication (distinct from a "no tasks available" result that makes the thread wait and try again). Threads terminate when they get that indication, and the overall pool management thread waits until all work threads have terminated and then terminates the pool. The main program has to wait until the pool is terminated and the pool management thread exits, once that happens it's safe to terminate the program. If the program needs to continue to run and restart the pool later, that's also the condition that must be met before it can do anything that would affect the pool configuration or restart the pool.