Difficulty calling unmanaged dll from C#

Question

I'm going bleary-eyed trying to figure out why I can't call an external method in an old C++ .dll from my C# application.

Here's the function header:

int __export FAR PASCAL SimplePGPEncryptFile(
                                      HWND hWnd1,
                                      LPSTR InputFileName,
                                      LPSTR OutputFileName,
                                      BOOL SignIt,
                                      BOOL Wipe,
                                      BOOL Armor,
                                      BOOL TextMode,
                                      BOOL IDEAOnly,
                                      BOOL UseUntrustedKeys,
                                      LPSTR RecipientList,
                                      LPSTR SignerKeyID,
                                      int SignerBufferLen,
                                      LPSTR SignerPassphrase,
                                      int SignerPwdBufferLen,
                                      LPSTR IDEAPassphrase,
                                      int IDEAPwdBufferLen,
                                      LPSTR PublicKeyRingName,
                                      LPSTR PrivateKeyRingName);

Here's my C# declaration:

        [DllImport("smplpgp_32.dll", CallingConvention = CallingConvention.StdCall)]
    public static extern int SimplePGPEncryptFile(
        IntPtr hWnd1,
        [MarshalAs(UnmanagedType.LPStr)] string InputFileName,
        [MarshalAs(UnmanagedType.LPStr)] string OutputFileName,
        bool SignIt,
        bool Wipe,
        bool Armor,
        bool TextMode,
        bool IDEAOnly,
        bool UseUntrustedKeys,
        [MarshalAs(UnmanagedType.LPStr)] string RecipientList,
        [MarshalAs(UnmanagedType.LPStr)] string SignerKeyID,
        int SignerBufferLen,
        [MarshalAs(UnmanagedType.LPStr)] string SignerPassphrase,
        int SignerPwdBufferLen,
        [MarshalAs(UnmanagedType.LPStr)] string IDEAPassphrase,
        int IDEAPwdBufferLen,
        [MarshalAs(UnmanagedType.LPStr)] string PublicKeyRingName,
        [MarshalAs(UnmanagedType.LPStr)] string PrivateKeyRingName);

When I call this method I'm getting one of the two following errors (declared in the header):

#define SIMPLEPGPENCRYPTFILE_RECIPIENTLISTDOESNOTENDWITHNEWLINE    408
#define SIMPLEPGPENCRYPTFILE_RECIPIENTLISTDOESNOTSTARTWITHGOODCODECHAR   409

This is also defined as a constant in the header:

#define INCLUDE_ONLYUSERIDS 1

This is C++ code that is known to work calling this function:

    char recipients[512];
recipients[0] = INCLUDE_ONLYUSERIDS;
strcat(strcpy(&recipients[1], rID), "\n"); \\ rID is equal to "CA"
return 0 == SimplePGPEncryptFile(INI.m_hWnd,
    (char*)plain, (char*)cipher,
    0,
    0,
    1,
    0,
    0,
    1, // UseUntrustedKeys
    recipients,
    0, 0,
    0, 0,
    0, 0,
    PGPKM.pub, 0); //PGPKM.pub is declared earlier

Passing this for the 'RecipientList' parameter gives me the '409' error:

string recipientList = "1CA\n\0";

Passing this for the 'RecipientList' parameter gives me the '408' error:

char[] recipients = new char[512];
recipients[0] = '1';
recipients[1] = 'C';
recipients[2] = 'A';
recipients[3] = '\n'; // also tried '\r', then '\n'
recipients[4] = Char.MinValue;
string paramValue = recipients.ToString();

Can anyone spot an obvious oversight on my part? I feel like I've got everything I need to solve this, but nothing is working as expected.

Side note: I'm successfully calling a different function in the same .dll. Also, I've experimented using StringBuilder to construct the RecipientList parameter.

Thanks for any suggestions!

Answer 1

Did You try to change the calling conventions? In C++ you declare the function with pascal calling convention and in C# as stdcall. The calling convention must match.

Answer 2

TLDR:

string recipientList = (char) 1 + "CA\n\0";

Did some digging based on Jeffora's comment... He's probably right, my first suggestion probably won't buy you anything. I wrote some test code, here's what I found:

When you use this code:

string recipientList = "1CA\n\0";

The bytes you end up with on the inside of the function after marshaling as an LPStr are:

49-67-65-10-0

Which looks like what you want.

When you use this code:

char[] recipients = new char[512];
recipients[0] = '1';
recipients[1] = 'C';
recipients[2] = 'A';
recipients[3] = '\n'; // also tried '\r', then '\n'
recipients[4] = Char.MinValue;
string paramValue = recipients.ToString();

The bytes you get on the inside of the function after marshaling as an LPStr are:

83-121-115-116-101-109-46-67-104-97-114-91-93

Which is ASCII for "System.Char[]". So you'll definitely need to use an encoder to turn that char[] into the string you want.

As for why the first string isn't working... I suspect your DLL is really looking for the value '1', and not the ascii character for '1' (49). In the code that works, you say:

recipients[0] = INCLUDE_ONLYUSERIDS;

where INCLUDE_ONLYUSERIDS = 1

Try this:

string recipientList = (char) 1 + "CA\n\0";

Original answer:
I believe the issue is that Char.ToString() treats the char as a unicode character, but you're marshalling as LPStr, not LPWStr. LPStr is looking for a string of ANSII characters.

To fix this, try using a byte array.

byte[] recipients= new byte[512];
 ...
//  Pass this string to SimplePGPEncryptFile
string recipientsToPass = System.Text.ASCIIEncoding.ASCII.GetString(recipients);