NumberOfConcurrentThreads parameter in CreateIoCompletionPort

Question

I am still confused about the NumberOfConcurrentThreads parameter within CreateIoCompletionPort(). I have read and re-read the MSDN dox, but the quote

This value limits the number of runnable threads associated with the completion port.

still puzzles me.

Question

Let's assume that I specify this value as 4. In this case, does this mean that:

1) a thread can call GetQueuedCompletionStatus() (at which point I can allow a further 3 threads to make this call), then as soon as that call returns (i.e. we have a completion packet) I can then have 4 threads again call this function,

or

2) a thread can call GetQueuedCompletionStatus() (at which point I can allow a further 3 threads to make this call), then as soon as that call returns (i.e. we have a completion packet) I then go on to process that packet. Only when I have finished processing the packet do I then call GetQueuedCompletionStatus(), at which point I can then have 4 threads again call this function.

See my confusion? Its the use of the phrase 'runnable threads'.

I think it might be the latter, because the link above also quotes

If your transaction required a lengthy computation, a larger concurrency value will allow more threads to run. Each completion packet may take longer to finish, but more completion packets will be processed at the same time.

This will ultimately affect how we design servers. Consider a server that receives data from clients, then echoes that data to logging servers. Here is what our thread routine could look like:

DWORD WINAPI ServerWorkerThread(HANDLE hCompletionPort)
{
    DWORD BytesTransferred;
    CPerHandleData* PerHandleData = nullptr;
    CPerOperationData* PerIoData = nullptr;

    while (TRUE)
    {
        if (GetQueuedCompletionStatus(hCompletionPort, &BytesTransferred,
            (PULONG_PTR)&PerHandleData, (LPOVERLAPPED*)&PerIoData, INFINITE))
        {
            // OK, we have 'BytesTransferred' of data in 'PerIoData', process it:
            // send the data onto our logging servers, then loop back around
            send(...);
        }
    }
    return 0;
}

Now assume I have a four core machine; if I leave NumberOfConcurrentThreads as zero within my call to CreateIoCompletionPort() I will have four threads running ServerWorkerThread(). Fine.

My concern is that the send() call may take a long time due to network traffic. Hence, I could be receiving a load of data from clients that cannot be dequeued because all four threads are taking a long time sending the data on?!

Have I missed the point here?

Update 07.03.2018 (This has now been resolved: see this comment.)

I have 8 threads running on my machine, each one runs the ServerWorkerThread():

DWORD WINAPI ServerWorkerThread(HANDLE hCompletionPort)
{
DWORD BytesTransferred;
CPerHandleData* PerHandleData = nullptr;
CPerOperationData* PerIoData = nullptr;

while (TRUE)
{
    if (GetQueuedCompletionStatus(hCompletionPort, &BytesTransferred,
        (PULONG_PTR)&PerHandleData, (LPOVERLAPPED*)&PerIoData, INFINITE))
    {
    switch (PerIoData->Operation)
    {
    case  CPerOperationData::ACCEPT_COMPLETED:
    {
        // This case is fired when a new connection is made
        while (1) {}
    }
}
}

I only have one outstanding AcceptEx() call; when that gets filled by a new connection I post another one. I don't wait for data to be received in AcceptEx().

I create my completion port as follows:

CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 4)

Now, because I only allow 4 threads in the completion port, I thought that because I keep the threads busy (i.e. they do not enter a wait state), when I try and make a fifth connection, the completion packet would not be dequeued hence would hang! However this is not the case; I can make 5 or even 6 connections to my server! This shows that I can still dequeue packets even though my maximum allowed number of threads (4) are already running? This is why I am confused!

Answer 1

the completion port - is really KQUEUE object. the NumberOfConcurrentThreads is corresponded to MaximumCount

Maximum number of concurrent threads the queue can satisfy waits for.

from I/O Completion Ports

When the total number of runnable threads associated with the completion port reaches the concurrency value, the system blocks the execution of any subsequent threads associated with that completion port until the number of runnable threads drops below the concurrency value.

it's bad and not exactly said. when thread call KeRemoveQueue ( GetQueuedCompletionStatus internal call it) system return packet to thread only if Queue->CurrentCount < Queue->MaximumCount even if exist packets in queue. system not blocks any threads of course. from another side look for KiInsertQueue - even if some threads wait on packets - it activated only in case Queue->CurrentCount < Queue->MaximumCount.

also look how and when Queue->CurrentCount is changed. look for KiActivateWaiterQueue (This function is called when the current thread is about to enter a wait state) and KiUnlinkThread. in general - when thread begin wait for any object (or another queue) system call KiActivateWaiterQueue - it decrement CurrentCount and possible (if exist packets in queue and became Queue->CurrentCount < Queue->MaximumCount and threads waited for packets) return packet to wait thread. from another side, when thread stop wait - KiUnlinkThread is called. it increment CurrentCount.

your both variant is wrong. any count of threads can call GetQueuedCompletionStatus(). and system of course not blocks the execution of any subsequent threads. for example - you have queue with MaximumCount = 4. you can queue 10 packets to queue. and call GetQueuedCompletionStatus() from 7 threads in concurrent. but only 4 from it got packets. another will be wait (despite yet 6 packets in queue). if some of threads, which remove packet from queue begin wait - system just unwait and return packet to another thread wait on queue. or if thread (which already previous remove packet from this queue (Thread->Queue == Queue) - so active thread) again call KeRemoveQueue will be Queue->CurrentCount -= 1;