CODEWITHSUNDEEP
c#c++dllimportmarshalling
patriot
patriot

Reputation: 107

Dll import Marshal C++ string issue

Here is the signature of the DLL function I am trying to import

typedef char*(*DLLFUNC_Encrypt)(const char*, unsigned int, unsigned int&);

Here is my C# code

[DllImport("AuthCrypto.dll", EntryPoint = "Encrypt")]
public static extern IntPtr Encrypt(IntPtr data, int size, ref int mode);

public static string EncryptData(string data)
{
    int mode = 5;
    IntPtr dataIntPtr = Marshal.StringToHGlobalAnsi(data);
    IntPtr data_enc = CryptoAPI.Encrypt(dataIntPtr, data.Length, ref mode);

    return Marshal.PtrToStringAnsi(data_enc);
}

Here is my exception:

A call to PInvoke function 'CryptoAPI::Encrypt' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.

Upvotes: 0

Views: 806

Answers (1)

Max Truxa
Max Truxa

Reputation: 3478

Looks like you need to specify the calling convention to use. Since on DLLFUNC_Encrypt none is set explicitly, the compiler uses its default. Which calling convention is considered being the default one depends on your compiler settings (for Visual Studio it is cdecl if you did not change it).

You can fix that by explicitly specifying the calling convention DLLFUNC_Encrypt like this:

typedef char*(*__cdecl DLLFUNC_Encrypt)(const char*, unsigned int, unsigned int&);

and adjusting your platform invoke accordingly:

[DllImport("AuthCrypto.dll", EntryPoint = "Encrypt", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr Encrypt(IntPtr data, int size, ref int mode);

The calling conventions you can pass to DllImport are defined in the CallingConvention enumeration.

I just read that the DLL you are invoking is closed source. If you can't specify the calling convention in the DLL, you can either look if you find the calling convention used by the library in its documentation or you can just test until you find a working one (it's probably either StdCall or Cdecl).

Upvotes: 1

Related Questions