IIS Native Module - Websocket

Question

I'm looking to implement a native websocket handler via a native module in IIS. I'm finding the documentation around it to be pretty vague and missing a lot of details.

I've created a IIS Native module DLL and it is working. I can keep it simple and just return a hello world html file with HTTP/200 response, and all is well.

However, I'm running into an issue when attempting to have it handle a websocket connection. There was a blog post on microsoft's IIS blog site here which describes how to handle a websocket connection. I've followed it and have tested and, yes, the connection is opened from a web browser, however I cannot read data from the socket in the native module, and the connection often appears to close in error on the client - after a random amount of time it seems.

The OnBeginRequest method of the module is:

REQUEST_NOTIFICATION_STATUS CKIIS::OnBeginRequest(IN IHttpContext* pHttpContext, IN IHttpEventProvider* pProvider) {
    UNREFERENCED_PARAMETER(pProvider);

    HRESULT hr;
    
    // I've only placed this here so I can attach a debugger.
    std::this_thread::sleep_for(std::chrono::seconds(10));

    this->_context = pHttpContext;

    IHttpResponse* pHttpResponse = pHttpContext->GetResponse();

    if (pHttpResponse != NULL)
    {
        pHttpResponse->Clear();

        pHttpResponse->SetStatus(101, "Switching Protocols");
        pHttpResponse->SetHeader(
            HttpHeaderUpgrade, "websocket",
            (USHORT)strlen("websocket"), TRUE);
        pHttpResponse->SetHeader(
            HttpHeaderConnection, "Upgrade",
            (USHORT)strlen("Upgrade"), TRUE);
        DWORD cbSent = 0;
        BOOL fCompletionExpected = false;
        hr = pHttpResponse->Flush(false, true, &cbSent, &fCompletionExpected);

        std::this_thread::sleep_for(std::chrono::seconds(10));

        IHttpContext3* pHttpContext3;
        HttpGetExtendedInterface(this->_server, pHttpContext, &pHttpContext3);

        IWebSocketContext* cts = (IWebSocketContext*)pHttpContext3->GetNamedContextContainer()->GetNamedContext(L"websockets");
    
        char buffer[1024 * 100];
        DWORD sz = 1024 * 100;
        BOOL utf;
        BOOL finalfrag;
        BOOL conclose;
        DWORD clxc = 78;
        BOOL expected;

        // This method call returns E_NOTIMPL.  
        // The documentation does not even indicate this is an expected return of this.
        HRESULT res = cts->ReadFragment(
            &buffer,
            &sz,
            false,
            &utf,
            &finalfrag,
            &conclose,
            Compl,
            &clxc,
            &expected);

        // Start a thread to read/write from the websocket.
        this->_runner = thread(&CKIIS::RunWork, this);

        // Tell IIS to keep the connection pending...
        return RQ_NOTIFICATION_PENDING;
    }

    // Return processing to the pipeline.
    return RQ_NOTIFICATION_CONTINUE;
}

void CKIIS::RunWork() {

    IHttpContext3* pHttpContext3;
    HttpGetExtendedInterface(this->_server, this->_context, &pHttpContext3);

    IWebSocketContext* cts = (IWebSocketContext*)pHttpContext3->GetNamedContextContainer()->GetNamedContext(L"websockets");
    
    for (;;) {
        // Loop to read/write the socket... 
        
        // If I call cts->ReadFragment() or cts->WriteFragment() here
        // the method will return E_NOTIMPL too.

        /// Eventually break out of the loop.
    }

    try {
        //this->_context->IndicateCompletion(RQ_NOTIFICATION_FINISH_REQUEST);
        this->_context->PostCompletion(0);
    }
    catch(const std::exception& e){
        const char* barf = e.what();
        std::cout << e.what();
    }
}

Few questions:

• Why does ReadFragment/WriteFragment return E_NOTIMPL. Both in the OnBeginRequest method, or if I attempt to do it in its own thread.
• Is using a new thread to handle the read/writing to the websocket the preferred pattern here? Having a busy loop like that polling for data seems pretty wasteful, but I cannot see any way to register for notifications when data comes available from the client

Answer 1

I have this working now, you'll want to do a few things:

• In regards to your polling the sending/receiving question, I found that with polling the receiving that putting in a Sleep(10) would cause IIS to make the w3wp.exe process go Suspended, polling with a Sleep(100) alleviated this issue. I try to make it a habit to put a Sleep() in when I need to do polling of some other thread(s) with a while loop.
• Read/WriteFragment are giving you E_NOTIMPL because they are not implemented apparently, I tried them and I either got E_NOTIMPL or it simply hang the w3wp.exe process. I don't know where you get DWORD cxlc = 78 because the completion expected is a void* that points to some IHttpCompletionInfo*, apparently... What I did instead was implement my own websocket messaging by reading RFC 6455. I'll leave that as an exercise to you because my crude implementation still needs some improvements such as properly chunking messages with taking into account the bytes of the header used for the payload and key sizes.
• Your response for the initial handshake needs to include the Sec-WebSocket-Accept header with the correct response key, which is the Base64 encoded SHA1 hash of the key as a string with 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 concatenated to the end of that string prior to hashing and encoding, see RFC 6455.
• I chose to use OnExecuteRequestHandler instead of OnBeginRequest and make a file format e.g. *.chatsocket that upgrades to a websocket. Ideally this would give me the ability to redirect the stdin/stdout/stderr of a console program to a websocket without writing a separate TCP server for each program when it could simply reside on the IIS server.
• However, this will hang until you kill the w3wp.exe process if you return RQ_NOTIFICATION_PENDING with OnExecuteRequestHandler, so what I did was implement a custom notification and run pHttpContext->NotifyCustomNotification(m_pCustomProvider, &expected) on each iteration. IIS hangs for whatever reason on RQ_NOTIFICATION_PENDING on an OnExecuteRequestHandler but not on OnCustomRequestNotification.
• Make sure to enable IHttpContext3::EnableFullDuplex() after the initial handshake and response Flush operation.
• Then to continue reading but don't rely on request->GetRemainingBytes() because it will continue to return 0.

I do something like this instead:

asyncReadLock.lock();
if (!isInAsyncRead) {
    isInAsyncRead = true;
    DWORD bytesReceived = 0;
    BOOL completionPending = false;
    HRESULT result = (*request)->ReadEntityBody(
        this->receiveBuffer,
        this->receiveBufferSize,
        true,
        &bytesReceived,
        &completionPending
    );
}
asyncReadLock.unlock();

Run ReadEntityBody but specify fAsync as true, and handle the asynchronous call like so:

REQUEST_NOTIFICATION_STATUS OnAsyncCompletion(
    IN IHttpContext* pHttpContext,
    IN DWORD dwNotification,
    IN BOOL fPostNotification,
    IN OUT IHttpEventProvider* pProvider,
    IN IHttpCompletionInfo* pCompletionInfo
)
{
    currentModule.asyncReadLock.lock();
    DWORD completedBytes = pCompletionInfo->GetCompletionBytes();
    HRESULT status = pCompletionInfo->GetCompletionStatus();
    if (completedBytes > 0) {
        for (int i = 0; i < (int)completedBytes; i++) {
            this->currentModule.ReceiveQueue.push_back(this->currentModule.receiveBuffer[i]);
        }
    }
    this->currentModule.isInAsyncRead = false;
    currentModule.asyncReadLock.unlock();
    return RQ_NOTIFICATION_PENDING;
}

I keep a std::deque<unsigned char> ReceiveQueue that keeps any bytes that come in via ReadEntityBody and in my main loop I'll pass those along to create WebSocketMessages as needed instead of using ReadFragment.

In summation, I've got a file handler now for IIS for a format like *.chatext or *.chatsocket that turns it into a websocket all in IIS as a Native Module without running a separate TCP server so I get the benefits of IIS features like HTTPS without the need for having to go through ARR or URL Rewrites, I have things working the way I expected them to work in the first place. I use ReadEntityBody instead of ReadFragment to continue reading incoming data after the handshake with asynchronous reads. I use read polling with a Sleep(100) on each poll, I also implement my own timeout for a set number of seconds to ensure IIS closes connections properly.

Let me know if you have any further questions.