Fastest way to send large blobs of data from one program to another in Windows?

Question

I need to send large blobs of data (~10MB) from one program to another in Windows 7. I would like a method that allows for at least a gigabyte per second total throughput with very low system load. To simplify this, all blobs may be the same size, and one program may be a child process of the other.

Method 1: Memory map the same file in both programs: CreateFileMapping() / MapViewOfFile()

In this case, the memory mapped file(s) presumably contains room for several blobs in a ring buffer. There would need to be some external mechanism to synchronize access to the ring buffer.

Method 2: Create named data sections

Method 3: WriteProcessMemory (suggested by Hristo Iliev below, thanks!)

Method 4: Read/write files on a RAM disk.

Method 5: Read/write to an anonymous pipe.

Method ?: Anything else? Perhaps write over TCP, use MPI, ...

I know that memory-mapped files (method 1) are considered the standard solution to this problem :)

• How fast are memory-mapped files? (rough order of magnitude)

• Is there an even faster method?

• How much worse is the performance of the other methods? Which ones of them can hit GB/sec throughput?

• If using memory mapped files, what is the best way for the programs to synchronize access to the data being passed? (ie: how would the producer indicate to the consumer that a new blob is available, and how would the consumer indicate it is done with a particular blob?)

• If using memory mapped files, is it better to have one file for all blobs together (ring buffer in a file), or one file for each blob (ring buffer of files)?

Answer 1

You could also use WriteProcessMemory and have the first process to directly post the data into the address space of the second process. You'd need to develop a protocol of some kind. For example, the second process could send the virtual address of its receive buffer to the first process via a named pipe or a shared memory block, then the first process copies the data using WriteProcessMemory and when it is finished, signals the second one via a semaphore or something. This ought to be the fastest way to send data between two processes as it involves a single copy operation. The first process would need to obtain the proper rights on the second one and that should not be a problem as long as both processes belong to the same user.