Migrating ctypes function from Python 2 to Python 3

Question

In case this is a XY problem, here is what i want to do:

I have a wxPython app, that has to communicate with another process using the WM_COPYDATA windows message. While sending the message with the ctypes module was surprisingly easy, receiving the answer requires me to overwrite the wx loop, since wx does not provide a specific event for this case.

On python2, I used the ctypes.windll.user32.SetWindowLongPtrW and the ctypes.windll.user32.CallWindowProcW Functions to get the desired behaviour. However, in python3, the same code leads to OSError: exception: access violation writing.

As far as I found out, the only difference between the python2 ctypes module and the python3 ctypes module is how they handle strings.

I also read, that there is a difference in how the two version layout the memory, but since I'm no C Expert, I can't find the problem in my code.

I have tested the code with python3.7 (64Bit) and python2.7(64Bit) and wx 4.0.7 (though it also works with wx2.8 and python2)

Here is minimal reproducible example:

import ctypes, ctypes.wintypes, win32con, wx, sys


_LPARAM = ctypes.wintypes.LPARAM
_WPARAM = ctypes.wintypes.WPARAM
_HWND = ctypes.wintypes.HWND
_UINT = ctypes.wintypes.UINT
_LPCWSTR = ctypes.wintypes.LPCWSTR
_LONG_PTR = ctypes.c_long
_LRESULT = _LONG_PTR
_LPCWSTR = ctypes.wintypes.LPCWSTR

_WNDPROC = ctypes.WINFUNCTYPE(_LPARAM,   # return Value
                              _HWND,     # First Param, the handle
                              _UINT,     # second Param, message id
                              _WPARAM,   # third param, additional message info (depends on message id)
                              _LPARAM,   # fourth param, additional message info (depends on message id)
)


_SetWindowLongPtrW = ctypes.windll.user32.SetWindowLongPtrW
_SetWindowLongPtrW.argtypes = (_HWND, ctypes.c_int, _WNDPROC)
_SetWindowLongPtrW.restypes = _WNDPROC

_CallWindowProc = ctypes.windll.user32.CallWindowProcW
_CallWindowProc.argtypes = (_WNDPROC, _HWND, _UINT, _WPARAM, _LPARAM)
_CallWindowProc.restypes = _LRESULT

def _WndCallback(hwnd, msg, wparam, lparam):
    print(hwnd, msg, wparam, lparam)
    return _CallWindowProc(_old_wndproc, hwnd, msg, _WPARAM(wparam), _LPARAM(lparam))
_mywndproc = _WNDPROC(_WndCallback)


app = wx.App(redirect=False)
frame = wx.Frame(None, title='Simple application')
frame.Show()

_old_wndproc = _WNDPROC( _SetWindowLongPtrW(frame.GetHandle(), win32con.GWL_WNDPROC, _mywndproc ) )
if _old_wndproc == 0:
    print( "Error" )
    sys.exit(1)

app.MainLoop()

Edit: I know that there is a pywin32 module, that could potentially help me. However, since the code works on python2 I'm rather curious what is going on here.

Answer 1

One problem is here:

_LONG_PTR = ctypes.c_long
_LRESULT = _LONG_PTR

The type LONG_PTR is "an integer the size of a pointer", which varies between 32-bit and 64-bit processes. Since you are using 64-bit Python, pointers are 64-bit and LONG_PTR should be:

_LONG_PTR = ctypes.c_longlong

If you want more portable code for 32- and 64-bit, LPARAM is also defined as LONG_PTR in the Windows headers so the below definition would define LONG_PTR correctly for 32-bit and 64-bit Python since ctypes already defines it correctly based on Python's build:

_LONG_PTR = ctypes.wintypes.LPARAM  # or _LPARAM in your case

After that change I tested your script with wxPython and still had an issue. I suspect wxPython is compiled without the UNICODE/_UNICODE definitions so the SetWindowLongPtr and CallWindowProc APIs must use the A version to retrieve and call the old window procedure. I made that change and the following code works.

Full code tested with 64-bit Python 3.8.8:
```py
import ctypes, ctypes.wintypes, win32con, wx, sys


_LPARAM = ctypes.wintypes.LPARAM
_WPARAM = ctypes.wintypes.WPARAM
_HWND = ctypes.wintypes.HWND
_UINT = ctypes.wintypes.UINT
_LPCWSTR = ctypes.wintypes.LPCWSTR
_LONG_PTR = _LPARAM
_LRESULT = _LONG_PTR
_LPCWSTR = ctypes.wintypes.LPCWSTR

_WNDPROC = ctypes.WINFUNCTYPE(_LRESULT,  # return Value
                              _HWND,     # First Param, the handle
                              _UINT,     # second Param, message id
                              _WPARAM,   # third param, additional message info (depends on message id)
                              _LPARAM,   # fourth param, additional message info (depends on message id)
)


_SetWindowLongPtr = ctypes.windll.user32.SetWindowLongPtrA
_SetWindowLongPtr.argtypes = (_HWND, ctypes.c_int, _WNDPROC)
_SetWindowLongPtr.restypes = _WNDPROC

_CallWindowProc = ctypes.windll.user32.CallWindowProcA
_CallWindowProc.argtypes = (_WNDPROC, _HWND, _UINT, _WPARAM, _LPARAM)
_CallWindowProc.restypes = _LRESULT

@_WNDPROC
def _WndCallback(hwnd, msg, wparam, lparam):
    print(hwnd, msg, wparam, lparam)
    return _CallWindowProc(_old_wndproc, hwnd, msg, wparam, lparam)


app = wx.App(redirect=False)
frame = wx.Frame(None, title='Simple application')
frame.Show()

_old_wndproc = _WNDPROC(_SetWindowLongPtr(frame.GetHandle(), win32con.GWL_WNDPROC, _WndCallback))
if _old_wndproc == 0:
    print( "Error" )
    sys.exit(1)

app.MainLoop()

As an aside, there is a note about SetWindowLongPtr (and similar for CallWindowProc) in the MSDN documentation that hinted at this solution:

The winuser.h header defines SetWindowLongPtr as an alias which automatically selects the ANSI or Unicode version of this function based on the definition of the UNICODE preprocessor constant. Mixing usage of the encoding-neutral alias with code that not encoding-neutral can lead to mismatches that result in compilation or runtime errors. For more information, see Conventions for Function Prototypes.