Async programming, threads and efficiency

Question

My question is not applicable only to C# and Asp.net but it is easier for me to ask more specifically.

When an Asp.net request awaits an async IO operation, the thread basically goes to the thread pool to be reused by other requests. Why is this more efficient than just sleeping the thread until the IO operation is finished? After all, when the thread is returned to the thread pool, its stack needs to be kept in memory to finish the original request. My assumption is that we cannot reuse the memory allocated to the thread unless we copy used stack memory somewhere else and copying data may introduce additional overhead that may not be justified.

Am I missing something? Are my assumtions wrong? Please explain.

EDIT: The answer pointed by jlew is missing one point. What happens to the stack memory used by the request when the thread is returned to the pool? If we cannot reuse the memory, then what is the point of reusing the thread? If we want to reuse the part of the stack that is not used, then we will have to move some memory around. If so, does moving memory around and reusing unused stack memory improve overall efficiency?

Answer 1

Why is this more efficient than just sleeping the thread until the IO operation is finished?

Think of threads as workers. Workers are expensive. Do you want to pay workers to sleep? No. You want to pay workers to do as much work as possible; if they're blocked, you get them working on something else rather than sleeping until the blockage is cleared up.

Threads are insanely expensive. If threads were cheap then sure, we could make lots. You only use pooling strategies for expensive resources.

Threads are expensive primarily in two ways: you mention the size of the stack, which is 1MB of reserved virtual memory per thread. But there are also significant costs at the OS level, which is not optimized for scenarios with thousands of threads. Determining which thread gets to run next, context switching to it, and switching away from it, all this has non-zero cost that increases as the number of threads increases. Ideally you want n independent threads in an n-processor machine, no more, no less.

After all, when the thread is returned to the thread pool, its stack needs to be kept in memory to finish the original request.

I can't make heads nor tails of this sentence. When the thread goes back to the pool its stack pointer is back at the bottom.

My assumption is that we cannot reuse the memory allocated to the thread unless we copy used stack memory somewhere else and copying data may introduce additional overhead that may not be justified.

I am beginning to make sense of it. Your mental model is:

• stack is the reification of continuation (what are we doing next?) and activation (what are the values of locals in this method activation?)
• awaited tasks that complete must continue at the point in the workflow where they left off, with the locals unchanged
• therefore the continuation/activation information -- the stack -- must be copied somewhere, or something.

This mental model is plausible but wrong. Asynchronous workflows build state machines and bundle up the continuation as states in that machine, and then store references to the state machine in the task. Activation information is hoisted from stack slots into fields of a closure class. This gets the continuation/activation information off the stack and into the heap.

Now, keep in mind that the continuation of a task does not contain all the information that is on the stack; it's not a true continuation in the "call with current continuation" sense that it captures what happens when the current method completes. It captures what happens when the currently awaited task completes, which is sufficient.

Remember, stacks can only be used as the reification of continuation when the "what am I going to do next?" workflow is logically a stack -- where the thing you do next is the thing on the top of the stack. But asynchronous workflows form a tree of dependencies where the join points are awaits; it's not a stack in the first place, so representing the continuations as a stack is a non-starter.

What happens to the stack memory used by the request when the thread is returned to the pool?

The million bytes of virtual memory reserved for a thread stack stays reserved as long as the thread continues to live. This is important to realize! This is one reason why threads are so expensive in .NET. That 1MB of virtual memory is 100% in use as far as the OS memory manager is concerned, no matter how much of that stack has been pushed to. Most of that 1MB is garbage for most of its lifetime.

The pointer to the high water mark of that stack gets moved back to the beginning when the thread goes back to the pool, and everything beyond it is now garbage.

If we cannot reuse the memory, then what is the point of reusing the thread?

We can and do reuse the memory; the stack pointer is reset.

If so, does moving memory around and reusing unused stack memory improve overall efficiency?

I'm not following the thrust of this question; I suspect it is based on faulty premises. Can you rephrase it?