CODEWITHSUNDEEP
c#debuggingwinapi
Johny Skovdal
Johny Skovdal

Reputation: 2104

Debugging error in calling SendMessage - GetLastWin32Error returns different errorcodes

I am trying to use the SendMessage API and debug why some calls seem to fail. I am starting to think that my original assumption that the API actually can return errors might be where I am going wrong though, which was the route I began with - getting the error code from the call, since something is going south. What I attempted in order to figure out how errors work, was to provide illegal input and see how the API reacts.

If I provide a negative handle, or a handle to a window that does not exist, the call returns false, and a call to Marshal.GetLastWin32Error() then returns 1400 (invalid window handle), which seems correct/expected.

When I then try to use the same boolean check on calling a window that should be able to process the message, but does not react to it, I get very different results depending on the context in which the code is executed (5 different scenarios total, repeated numerous times with consistent results), I get 3 different error codes:

  1. Running with debugger attached in Visual Studio - Debug: Returns the error code 0.
  2. Running with debugger attached in Visual Studio - Debug: Returns the error code 0.
  3. Running from commandline - Debug: Returns the error code 1150 (token does not exist).
  4. Running from commandline - Release: Error code gone again, and returns 0.
  5. Single stepping in VS and then use Immediate Window for error code retrieval - Debug: Returns the error code 1008 (newer windows version required).
  6. Single stepping in VS and then use Immediate Window for error code retrieval - Release: I'm unable to get the debugger to attach in that scenario, possibly due to wrongly configured VS debug settings, but I see that as a minor issue compared to the different results I get above.

The only error code above that seems reasonably correct to me is the 5th scenario, with the error code 1008, because the handle might not be accessible from the calling context, which is the scenario I am trying to debug.

Sample code

The code snippet I am testing looks like this:

if (!SendMessage(new IntPtr(windowHandleId), WM_COPYDATA, 0, ref copyDataStruct))
{
  var win32Error = Marshal.GetLastWin32Error();

and SendMessage is defined like this:

[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool SendMessage(IntPtr windowHandle, int message, int wordParam, ref COPYDATASTRUCT longParam);

Where am I going wrong in my setup, and how do I make sure I get the correct error?

I get a meaningful error when using invalid window handles, so my assumption is that setting the error code is supported by the SendMessage API. Is that the assumption that is incorrect though?

Upvotes: 0

Views: 1619

Answers (1)

David Heffernan
David Heffernan

Reputation: 613262

The documentation for SendMessage says:

The return value specifies the result of the message processing; it depends on the message sent.

In other words, we must refer to the documentation of WM_COPYDATA. It states:

If the receiving application processes this message, it should return TRUE; otherwise, it should return FALSE.

At no point in this is there any statement concerning GetLastError. Which means that you cannot expect GetLastError to yield an informative value. You'll just get whatever was passed in the most recent call to SetLastError, quite possibly a call made before your call to SendMessage.

All that you can infer from a value of false being returned from SendMessage is that the receiving application did not process the message.

Upvotes: 2

Related Questions