C++ and C# Communication using Named Pipe

Question

I'm trying to reverse enginering a dll injected into a process, that does hook winsock send() and send data over a PipeStream.

This is the C# code that read the pipe stream:

    [StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
    public struct PipeHeader
    {
        [MarshalAs(UnmanagedType.I1)]
        public byte command;
        [MarshalAs(UnmanagedType.I4)]
        public int sockid;
        public int datasize;
    }

    public static object RawDeserializeEx(byte[] rawdatas, Type anytype)
    {
        int rawsize = Marshal.SizeOf(anytype);
        if (rawsize > rawdatas.Length)
            return null;
        GCHandle handle = GCHandle.Alloc(rawdatas, GCHandleType.Pinned);
        IntPtr buffer = handle.AddrOfPinnedObject();
        object retobj = Marshal.PtrToStructure(buffer, anytype);
        handle.Free();
        return retobj;
    }
    private void PipeRead()
    {
        byte[] dbPipeMsgIn = new byte[9];
        byte[] zero = new byte[] { 0 };

        byte[] dbPipeMsgInData;
    PipeLoop:
        while (pipeIn.Read(dbPipeMsgIn, 0, 9) != 0)
        {
            strPipeMsgIn = (PipeHeader)RawDeserializeEx(dbPipeMsgIn, typeof(PipeHeader));
            if (strPipeMsgIn.datasize != 0)
            {
                dbPipeMsgInData = new byte[strPipeMsgIn.datasize];
                pipeIn.Read(dbPipeMsgInData, 0, dbPipeMsgInData.Length);
                //do something with dbPipeMsgInData
            }
        }
        if (pipeIn.IsConnected) goto PipeLoop;
    }

So far i have hooked the send() func, connect and send messages trough the pipe. The problem is that the data received is not the data i expect, so probably I do sent in a wrong way. I need help because i have very little to no C++ knowledge.

C++ code:

#pragma pack(1)
typedef struct
{
    byte command;
    int sockid;
    int datasize;
} PipeHeader;
#pragma pack(0)

int WINAPI MySend(SOCKET s, const char* buf, int len, int flags)
{
    PipeHeader ph;
    string p(buf);
    if(p.find("<TalkMsg") == 0){
        byte cmd = 0;
        ph.command = cmd;
        ph.sockid = s;
        ph.datasize = len;
        char buffer[sizeof(ph)];
        memcpy(buffer, &ph, sizeof(ph)); 
        if(SendPipeMessage(buffer, sizeof(buffer))){
            if(SendPipeMessage(buf, sizeof(buf))){
                MessageBox(NULL,"Message Sent", NULL, NULL);
            }
        }
        fopen_s(&pSendLogFile, "C:\\SendLog.txt", "a+");
        fprintf(pSendLogFile, "%s\n", buf);
        fclose(pSendLogFile);
    }
    return pSend(s, buf, len, flags);
}

BOOL SendPipeMessage(LPCVOID lpvMessage, DWORD ctToWrite){
    // Send a message to the pipe server. 
   cbToWrite = sizeof(lpvMessage);

   fSuccess = WriteFile( 
      hPipe,                  // pipe handle 
      lpvMessage,             // message 
      cbToWrite,              // message length 
      &cbWritten,             // bytes written 
      NULL);                  // not overlapped 

   if ( ! fSuccess) 
   {
      return false;
   }else return true;
}

Edit, more info:

The goal is to send a message containing a 9 Bytes long PipeHeader Struct over the pipe, then send another message containing the winsock send() data, (the buf variable), read the pipe on the C# application and parse the first message to get the DataSize of the next incoming message (that's the datasize var of PipeHeader), then, using the datasize read again the pipe to get the send() buffer. I think that's working in that way. I do not know very well how Pipes work.

Anyway the MAIN goal is to send the send() buffer from the C++ Dll into the C# application.

Update: Seems that i have to first serialize the PipeHeader struct in a way so i can Deserialize it using RawDeserializeEx() in the C# code. I tried by doing:

char buffer[sizeof(ph)];
memcpy(buffer, &ph, sizeof(ph)); 

The problem is that, in C++ sizeof(ph), or sizeof(buffer) return 12 bytes. Instead in C# the unmanaged size (Marshal.SizeOf()) of the same Struct, return 9 bytes.

Update 2: Solved the size differences by changing the struct packing. But I'm still not getting the right values in C#. Code Updated.

Answer 1

Your question wasn't particularly clear to me. I wasn't sure if the problem was handling the pipes or the serialization. I've provided below full C++ pipe server code and C# pipe client code. I've tried to keep things as simple as is useful to demonstrate a dialog. I used your struct and deserialization code and added a serialization method. In both code samples there is a link to MS doc that should make it easy to reverse the client server relationship, if desired. The server and client code are stand alone and designed to run in different processes (two console applications). There is no threading or and the server will not handle more than one connection.

In terms of simple (de/)serialization I would suggest you use the BitConverter class and methods inside the struct to manually convert to and from the byte array. This gives you a finer level of control and is easier to debug. If you have complex structs to represent I would suggest you look at Google's 'protobuf'. Of course YMMV.

C++ pipe server code (Windows)

#include <stdio.h>
#include <stdint.h>
#include <winsock.h>
#include <string>

// see https://msdn.microsoft.com/en-us/library/bb546085(v=vs.110).aspx?cs-save-lang=1&cs-lang=csharp#code-snippet-1
// for information on creating pipe clients and servers in c++

using namespace std;

#pragma pack(1)
typedef struct {
    int8_t command;
    int32_t sockid;
    int32_t datasize;
    } PipeHeader;
#pragma pack()

int wmain( int argc, wchar_t* argv[] ) {
    auto pipename = "\\\\.\\pipe\\pipey"; // can be any name, but must start '\\.\pipe\'
    printf("Create pipe '%s'\r\n", pipename);
    auto pipeHandle = CreateNamedPipe(pipename, 
                                      PIPE_ACCESS_DUPLEX, 
                                      PIPE_TYPE_MESSAGE | PIPE_WAIT,
                                      1,  // # instances
                                      64, // out buff
                                      64, // in buff
                                      0,  // timeout, 0 = default of 50ms
                                      NULL); // security attrs
    printf("Waiting for pipe connect\r\n");
    if (ConnectNamedPipe(pipeHandle, NULL)) {
        printf("Pipe connected\r\n");
        PipeHeader hdr;
        DWORD bytesRead;
        int8_t cmd = -1;
        while (cmd != 0) {
            if (ReadFile(pipeHandle, &hdr, sizeof(PipeHeader), &bytesRead, NULL)) {
                // you can check or assert here that bytesRead == sizeof(PipeHeader)
                printf("Read %d bytes from pipe, {command: %d, sockid: %d, datasize: %d}\r\n", 
                       bytesRead, hdr.command, hdr.sockid, hdr.datasize);
                if (hdr.command == 0) { // assume 0 is the exit cmd
                    break;
                    }
                else if (hdr.command == 1) {
                    // do whatever cmd 1 is ...
                    }
                hdr.command = 4;
                hdr.sockid = 101;
                hdr.datasize *= 2;
                if (WriteFile(pipeHandle, &hdr, sizeof(PipeHeader), &bytesRead, NULL)) {
                    printf("Data written to pipe\r\n"); 
                    }
                else {
                    printf("Pipe write failed\r\n");
                    break;
                    }
                }
            else {
                printf("Read error: %d\r\n", GetLastError());
                break; // exit
                }
            }
        if (DisconnectNamedPipe(pipeHandle) == FALSE) {
            printf("Disconnect error: %d\r\n", GetLastError());
            }
        }
    else {
        printf("Pipe connect failed\r\n");
        }
    CloseHandle(pipeHandle);
    printf("Exiting program\r\n");
    return 0;
    }

C# client pipe code (Windows)

using System;
using System.Runtime.InteropServices;
using System.IO.Pipes;

namespace Pipes {
    // see https://msdn.microsoft.com/en-us/library/bb546085(v=vs.110).aspx?cs-save-lang=1&cs-lang=csharp#code-snippet-1
    // for information on creating pipe clients and servers in c#
    class Program {
        [StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
        public struct PipeHeader {
            [MarshalAs(UnmanagedType.I1)]
            public byte command;
            [MarshalAs(UnmanagedType.I4)]
            public Int32 sockid;
            public Int32 datasize;
        }

        static void Main(string[] args) {
            var pipename = "pipey";
            var pipeClient = new NamedPipeClientStream(pipename);
            Console.WriteLine("Connecting to server pipe '{0}'", pipename);
            pipeClient.Connect();
            var hdr = new PipeHeader();
            var hdrSize = Marshal.SizeOf(hdr);
            hdr.command = 1;
            hdr.sockid = 1912;
            hdr.datasize = 32;
            var buf = Serialize(hdr);
            Console.WriteLine("Writing to server pipe");
            pipeClient.Write(buf, 0, hdrSize);
            pipeClient.Read(buf, 0, hdrSize);
            hdr = (PipeHeader) Deserialize(buf, hdr.GetType());
            Console.WriteLine("Pipe read {{ command: {0}, sockid: {1}, datasize: {2} }}",
                               hdr.command, hdr.sockid, hdr.datasize);
            hdr.command = 0; // tell server to disconnect
            buf = Serialize(hdr);
            Console.WriteLine("Sending disconnect");
            pipeClient.Write(buf, 0, hdrSize);
            pipeClient.Close();
            Console.WriteLine("Pipe closed");
            }

        public static object Deserialize(byte[] rawdatas, Type anytype) {
            int rawsize = Marshal.SizeOf(anytype);
            if (rawsize > rawdatas.Length)
                return null;
            GCHandle handle = GCHandle.Alloc(rawdatas, GCHandleType.Pinned);
            IntPtr buffer = handle.AddrOfPinnedObject();
            object retobj = Marshal.PtrToStructure(buffer, anytype);
            handle.Free();
            return retobj;
            }

        public static byte[] Serialize(object obj) {
            int rawsize = Marshal.SizeOf(obj);
            var rv = new byte[rawsize];
            IntPtr ptr = Marshal.AllocHGlobal(rawsize);
            Marshal.StructureToPtr(obj, ptr, true);
            Marshal.Copy(ptr, rv, 0, rawsize);
            Marshal.FreeHGlobal(ptr);
            return rv;
            }
        }
    }

Compile both sets of code. Start the server code, then start the client. Output is somewhat minimal, but the exchange of data is shown and the server modifies the data before sending it back (showing a two sided data exchange). There is no real purpose other than to serve as an example.

Hopefully, this example will provide the basic information you need to solve you problem.

Answer 2

Didn't check the whole code as I'm out of time right now. But what I've noticed:

• I'm pretty sure you can't expect that an int is always 4 bytes in C++. Where size matters an int is not the best choice.
• You forgot a [MarshalAs(UnmanagedType.I4)] for the member datasize in your C# code. Attributes are only applied to the next item, not to all following items.
• Replace SendPipeMessage(buffer, sizeof(buffer)) with SendPipeMessage((char*)&ph, sizeof(ph)) to get the desired result. No need for memcpy to a char array there and as M.L. mentioned sizeof(buffer) is not what you want.
• Replace sizeof(buf) with len. You already have the length so why use sizeof? And sizeof(buf) is again not what you want.
• You can simplify your return code at the end to return fSuccess;. No need for if-else there. Or just return WriteFile(...);.
• Why you use cbToWrite = sizeof(lpvMessage);? This again is not what you want and you already pass ctToWrite which you should use instead in the call of WriteFile.

Answer 3

My first post, so go easy on me please :-)

BOOL SendPipeMessage(LPCVOID lpvMessage){
    // Send a message to the pipe server. 
   cbToWrite = sizeof(lpvMessage);

sizeof is used incorrectly. It will return size of type LPCVOID instead of size of buffer as you plan. You're sending 4 bytes (on x86), when you want to send 9 bytes.

In my opinion, you need to provide new parameter for your SendPipeMessage()

BOOL SendPipeMessage(LPCVOID lpvMessage, DWORD cbToWrite)

You can use sizeof in first call. On second call you may use len as argument.

if(SendPipeMessage((LPCVOID)&ph, sizeof(ph))){
    if(SendPipeMessage(buf, len)){