Using and Overlapped Named Pipe in Python 3.7

Question

I am using Windows with Python 3.7 and I am trying to asynchronly share data, ony strings, between to python processes. One of the is running indefinitely (reciever), the other one may start at any point sends some data and then ends (sender). I am trying to use a named pipe for that.

I managed to get this when they run synchronously (the reciever waiting at an blocked pipe until he gets data), however the reciever has other stuff to do so ideally he shouldn't wait all the time. Also there might be a second sender at some point so a blocked pipe isin't great.

The code for the receiver is:

import os
import time
import sys
import win32pipe, win32file, pywintypes

pipe_name = r'\\.\pipe\mypipe' 

pipe = win32pipe.CreateNamedPipe(
        pipe_name,
        win32pipe.PIPE_ACCESS_DUPLEX | win32file.FILE_FLAG_OVERLAPPED,  # open mode 
        win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE | win32pipe.PIPE_WAIT, # pipe mode
        1, 65536, 65536, # max instances, out buffer size,  in buffer size
        0, # timeout
    None)


while 1:
    print("doing my other stuff")
    try:
        win32pipe.ConnectNamedPipe(pipe, pywintypes.OVERLAPPED())
    except pywintypes.error as e:
        if e.winerror == 232:   #disconnected pipe
            win32pipe.DisconnectNamedPipe(pipe)
            print("Disconnecting pipe")
        else:
            print(e)
    try:
        retval, msg = win32file.ReadFile(pipe, 0, pywintypes.OVERLAPPED())
        print(msg)
    except pywintypes.error as e:
        if e.winerror == 536: #Wating for connection
            print("waiting")
        elif e.winerror == 233: #no process on other side
            continue
    time.sleep(1)

The code for the sender is:

import os
import time
import sys
import win32pipe, win32file, pywintypes


pipe_name = r'\\.\pipe\mypipe' 


for x in range(5):

    handle = win32file.CreateFile(
            pipe_name,
            win32file.GENERIC_READ | win32file.GENERIC_WRITE,
            0,
            None,
            win32file.OPEN_EXISTING,
            win32file.FILE_FLAG_OVERLAPPED,
            None
            )
    res = win32pipe.SetNamedPipeHandleState(handle, win32pipe.PIPE_READMODE_MESSAGE, None, None)
    print(f"sending {x}")
    win32file.WriteFile(handle, str.encode(f"hello world {x}"))
    win32file.CloseHandle(handle)
    time.sleep(2)

Right now both can run and have some connection but i cant really get the data. The reciever can do oter stuff and disconnects and reopens the pipe if something is send, but msg ends up being empty. If i stop it in the debugger and send something the value of msg gets "memory at 0x0......." which I would interpret as some sort of pointer, yet as you probalby already noticed my understaing of pipes is limited.

Here I found a good example for a synchronos pipe which worked. I changed the the creation of the pipe to the reciever but that wasn't to hard. I found some examples for asynchronous (overlapped) pipes here, which were also great but left me with the issue i am facing now.

Ist reading from an overlapped pipe still a task for win32file.ReadFileor is their something else I am missing?

Thank you very much!

Answer 1

The poster's code and accepted answer do not handle ERROR_IO_PENDING. Unfortunately I could not find a good Python example anywhere else so I spent some time working it out.

Modify for your environment, connect to your pipe server, and call receive(). It will return a full message from the pipe, or None.

If you want to watch how the code works, set the size of read_buf to a low value such as 64 bytes and add some logging statements.

The same code should work equally well (with lite modification) for files and sockets.

import win32file
import win32pipe
import winerror
import pywintypes

class pipe_comms:
    def __init__(self):
        self.pipe_open = False
        self.pipe_handle = None
        self.pipe_name = r”\\.\pipe\your_pipe”
        self.read_msg = None
        self.overlapped = None
        self.read_buf = win32file.AllocateReadBuffer(4096)

    def connect(self):
        if not self.pipe_open:
            try:
                self.pipe_handle = win32file.CreateFile(self.pipe_name
                    win32file.GENERIC_READ | win32file.GENERIC_WRITE,
                    0, None, win32file.OPEN_EXISTING,
                    win32file.FILE_FLAG_OVERLAPPED, None)
                win32pipe.SetNamedPipeHandleState(self.pipe_handle, 
                    win32pipe.PIPE_READMODE_MESSAGE, None, None)
                self.pipe_open = True
                self.read_msg = None
            except pywintypes.error as e:
                self.handle_error(e.args[0])
                return False
        return True

    def receive(self):
        try:
            result = self.receive_overlapped()
            if result == winerror.ERROR_SUCCESS:
                fullMsg = self.read_msg
                self.read_msg = None
                return fullMsg
            elif result == winerror.ERROR_IO_PENDING:
                return None
            else:
                self.handle_error(result)
        except pywintypes.error as e:
            self.handle_error(e.args[0])
            return None

    def receive_overlapped(self):
        if self.overlapped:
            try:
                bytes_read = win32file.GetOverlappedResult(self.pipe_handle, self.overlapped, 0)
                self.read_msg += bytes(self.read_buf[:bytes_read])
                self.overlapped = None
                return winerror.ERROR_SUCCESS
            except pywintypes.error as e:
                result = e.args[0]
                if result == winerror.ERROR_MORE_DATA:
                    bytes_read = len(self.read_buf)
                    self.read_msg += bytes(self.read_buf[:bytes_read])
                    # fall thru to issue new ReadFile
                else:
                    # ERROR_IO_PENDING is normal, anything else is not
                    return result 
        else:
            self.read_msg = bytes()
            self.overlapped = pywintypes.OVERLAPPED()

        result, data = win32file.ReadFile(self.pipe_handle, self.read_buf, self.overlapped)
        while True:
            if result = winerror.ERROR_MORE_DATA:
                bytes_read = len(pipe_data)
                self.read_msg = bytes(data[:bytes_read])
                result, data = win32file.ReadFile(self.pipe_handle, self.read_buf, self.overlapped)
                continue
            elif result == winerror.ERROR_SUCCESS:
                bytes_read = win32file.GetOverlappedResult(self.pipe_handle, self.overlapped, 0)
                self.read_msg = bytes(data[:bytes_read])
                self.overlapped = None
            return result

    def handle_error(self, result):
        reset_pipe = False
        if result == winerror.ERROR_BROKEN_PIPE:
            win32pipe.DisconnectNamedPipe(self.pipe_handle)
            reset_pipe = True
        elif result == winerror.ERROR_NO_DATA:
            reset_pipe = True
            
        if reset_pipe:
            self.pipe_handle = None
            self.pipe_open = False
            self.read_msg = None
            self.overlapped = None
        

Answer 2

I found the solution and want to share it, just in case anyone else is stumbeling upon this issue. As it turns out msg gets "memory at 0x0......." is a memoryview object and its data can be exposed via bytes(msg).

also there was an issue with my ReadFile command as the buffer has to >0 for it to achieve anything. Now it reads every byte individually and adds them up to a string. This is proabably not great performancewise, but it works for me and it solves the issue of having to cut the end of if the message is shorter than the buffer length.

msg = ''
rtnvalue, data = win32file.ReadFile(pipe, 1, pywintypes.OVERLAPPED())
while rtnvalue == 234:
    msg = msg + bytes(data).decode('ASCII') 
    rtnvalue, data = win32file.ReadFile(pipe, 1, pywintypes.OVERLAPPED())
if rtnvalue == 0: #end of stream is reached
    msg = msg + bytes(data).decode('ASCII')
return msg