How do I write a custom marshaler which allows data to flow from native to managed?

Question

In attempting to write a custom marshaler related to this question (P/Invoke from C to C# without knowing size of array), I have come across something I cannot understand. This is the first ever custom marshaler that I have written so no doubt I'm missing something obvious due to my ignorance.

Here's my C# code:

using System;
using System.Runtime.InteropServices;
using System.Text;

namespace CustomMarshaler
{
    public class MyCustomMarshaler : ICustomMarshaler
    {
        static MyCustomMarshaler static_instance;

        public IntPtr MarshalManagedToNative(object managedObj)
        {
            if (managedObj == null)
                return IntPtr.Zero;
            if (!(managedObj is int[]))
                throw new MarshalDirectiveException("VariableLengthArrayMarshaler must be used on an int array.");

            int[] arr = (int[])managedObj;
            int size = sizeof(int) + arr.Length * sizeof(int);
            IntPtr pNativeData = Marshal.AllocHGlobal(size);
            Marshal.WriteInt32(pNativeData, arr.Length);
            Marshal.Copy(arr, 0, pNativeData + sizeof(int), arr.Length);
            return pNativeData;
        }

        public object MarshalNativeToManaged(IntPtr pNativeData)
        {
            int len = Marshal.ReadInt32(pNativeData);
            int[] arr = new int[len];
            Marshal.Copy(pNativeData + sizeof(int), arr, 0, len);
            return arr;
        }

        public void CleanUpNativeData(IntPtr pNativeData)
        {
            Marshal.FreeHGlobal(pNativeData);
        }

        public void CleanUpManagedData(object managedObj)
        {
        }

        public int GetNativeDataSize()
        {
            return -1;
        }

        public static ICustomMarshaler GetInstance(string cookie)
        {
            if (static_instance == null)
            {
                return static_instance = new MyCustomMarshaler();
            }
            return static_instance;
        }
    }
    class Program
    {
        [DllImport(@"MyLib.dll")]
        private static extern void Foo(
            [In, Out, MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(MyCustomMarshaler))]
            int[] arr
        );

        static void Main(string[] args)
        {
            int[] colorTable = new int[] { 1, 2, 3, 6, 12 };
            Foo(colorTable);
            foreach (int value in colorTable)
                Console.WriteLine(value);
        }
    }
}

On the other side is a trivial native DLL, written in Delphi as it happens.

library MyLib;

procedure Foo(P: PInteger); stdcall;
var
  i, len: Integer;
begin
  len := P^;
  Writeln(len);
  for i := 1 to len do begin
    inc(P);
    Writeln(P^);
    inc(P^);
  end;
end;

exports
  Foo;

begin
end.

The idea is that the array is passed to the DLL which then prints out the length field, and the values of the array. The native code also increments each value of the array by 1.

So, I expect to see this output:

5
1
2
3
6
12
2
3
4
7
13

But unfortunately I see this output:

5
1
2
3
6
12
1
2
3
6
12

Under the debugger I can see that MarshalNativeToManaged is executing, and that the values that it returns have been incremented. But these incremented values don't find there way back into the object that is passed to Foo.

What do I need to do to fix this?

Answer 1

Thanks a lot to Stephen and Hans for their excellent answers. I can see clearly now that I must keep hold of the managed object that was passed to MarshalManagedToNative, and then return the same object from the subsequent call to MarshalNativeToManaged.

It's a bit of a bind that the framework provides no mechanism for managing such state. That's because the marshaller uses the same instance of the custom marshaller for each call to the function.

Stephen's approach of using thread local storage will work I believe. I'm personally not a fan of thread local storage. Another option is to use a dictionary keyed on the unmanaged pointer. Here's an illustration:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Text;

namespace CustomMarshaler
{
    public class MyCustomMarshaler : ICustomMarshaler
    {
        private Dictionary<IntPtr, object> managedObjects = new Dictionary<IntPtr, object>();

        public IntPtr MarshalManagedToNative(object managedObj)
        {
            if (managedObj == null)
                return IntPtr.Zero;
            if (!(managedObj is int[]))
                throw new MarshalDirectiveException("MyCustomMarshaler must be used on an int array.");

            int[] arr = (int[])managedObj;
            int size = sizeof(int) + arr.Length * sizeof(int);
            IntPtr pNativeData = Marshal.AllocHGlobal(size);
            Marshal.WriteInt32(pNativeData, arr.Length);
            Marshal.Copy(arr, 0, pNativeData + sizeof(int), arr.Length);

            lock (managedObjects)
            {
                managedObjects.Add(pNativeData, managedObj);
            }

            return pNativeData;
        }

        public object MarshalNativeToManaged(IntPtr pNativeData)
        {
            int[] arr;
            lock (managedObjects)
            {
                arr = (int[])managedObjects[pNativeData];
                managedObjects.Remove(pNativeData);
            }
            int len = Marshal.ReadInt32(pNativeData);
            Debug.Assert(len == arr.Length);
            Marshal.Copy(pNativeData + sizeof(int), arr, 0, len);
            return arr;
        }

        public void CleanUpNativeData(IntPtr pNativeData)
        {
            Marshal.FreeHGlobal(pNativeData);
        }

        public void CleanUpManagedData(object managedObj)
        {
        }

        public int GetNativeDataSize()
        {
            return -1;
        }

        public static ICustomMarshaler GetInstance(string cookie)
        {
            return new MyCustomMarshaler();
        }
    }

    class Program
    {
        [DllImport(@"MyLib.dll")]
        private static extern void Foo(
            [In, Out, MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(MyCustomMarshaler))]
            int[] arr
        );

        static void Main(string[] args)
        {
            int[] colorTable = new int[] { 1, 2, 3, 6, 12 };
            Foo(colorTable);
            foreach (int value in colorTable)
                Console.WriteLine(value);
        }
    }
}

Answer 2

I had a similar problem many years ago and found that there was very little documentation on Custom Marshaling. I suspect using ICustomMarshaler never really took off since it can always be done using manual marshaling in the course of your regular code. And so there was never really a need for any documentation of advanced custom marshaling scenarios.

Anyway, through a variety of sources and much trial and error I think I teased out a practical understanding of how most of Custom Marshaling works.

In your case, you have set up the ManagedToNative method correctly for [In] marshaling and the NativeToManaged method correctly for most [Out] marshaling but [In, Out] marshaling is actually a bit trickier. [In, Out] marshaling is actually in-place marshaling. So on the way back out you must marshal the data back to the same instance that was provided in the [In] side of the operation.

There are a number of small variations on this depending on whether using reference or value types, whether the call is a normal pInvoke call or a callback on a delegate, etc. But thinking about what needs to end up where is the key.

The following variation on your code works the way you want it to (and it seems to works the same way for .Net 2.0 and up):

        //This must be thread static since, in theory, the marshaled
    //call could be executed simultaneously on two or more threads.
    [ThreadStatic] int[] marshaledObject;

    public IntPtr MarshalManagedToNative(object managedObj)
    {
        if (managedObj == null)
            return IntPtr.Zero;
        if (!(managedObj is int[]))
            throw new MarshalDirectiveException("VariableLengthArrayMarshaler must be used on an int array.");

        //This is called on the way in so we must keep a reference to 
        //the original object so we can marshal to it on the way out.
        marshaledObject = (int[])managedObj;
        int size = sizeof(int) + marshaledObject.Length * sizeof(int);
        IntPtr pNativeData = Marshal.AllocHGlobal(size);
        Marshal.WriteInt32(pNativeData, marshaledObject.Length);
        Marshal.Copy(marshaledObject, 0, (IntPtr)(pNativeData.ToInt64() + sizeof(int)), marshaledObject.Length);
        return pNativeData;
    }

    public object MarshalNativeToManaged(IntPtr pNativeData)
    {
        if (marshaledObject == null)
            throw new MarshalDirectiveException("This marshaler can only be used for in-place ([In. Out]) marshaling.");

        int len = Marshal.ReadInt32(pNativeData);
        if (marshaledObject.Length != len)
            throw new MarshalDirectiveException("The size of the array cannot be changed when using in-place marshaling.");

        Marshal.Copy((IntPtr)(pNativeData.ToInt64() + sizeof(int)), marshaledObject, 0, len);

        //Reset to null for next call;
        marshalledObject = null;

        return marshaledObject;
    }

Answer 3

    int len = Marshal.ReadInt32(pNativeData);
    int[] arr = new int[len];

Your problem is located here, you are creating a new array. But you need to update the colorTable array instead. You got a reference to it in your MarshalManagedToNative() method, you'll need to store it so you can use it again in your MarshalNativeToManaged() method.

Note that this has a number of consequences. Your custom marshaller object becomes stateful, you can't use a static instance anymore. And you'll need a different approach if the unmanaged code modifies the array length. You allow for this by reading the len back but didn't actually implement it, so it's okay. Just assert that the length didn't change.