When using Pinvoke to pass an LPBYTE

Question

I have the delightful (and by delightful I mean painstakingly tedious) job of converted a VB6 application to VB.net. One of the things this application does is communicate with c dll’s (30 of them). After writing a number of the structures and declarations by hand, I decided an automated tool would help save time so I purchased the pinvoke wizard tool.

I ran it threw on of the header files and it successfully generated VB code but I do not understand how to use it, let me give you an example. One of the methods I call is StartSession. In C the header code is

WORD APIENTRY StartSession 
(LPBYTE lpbReturnCode,
LPBYTE lpbErrorMessage,
LPBYTE lpbApplName,
LPBYTE lpbRegionName,
LPBYTE lpbSessionID, 
LPBYTE lpbDataTrunc,
LPBYTE lpbSessionToken,
LPBYTE lpbStandardWait,
LPBYTE lpbVersion,
LPBYTE lpbEnvironment,
LPBYTE lpbESILogonOnly,
LPBYTE lpbMetadataSource,
LPWORD lpbMetadataCacheCount); 

When I rewrote this function by hand I had written it as

Declare Function StartSession Lib "chat2.dll" (ByVal ReturnCode As String _
                                               , ByVal ErrorMsg As String _
                                               , ByVal ApplicationName As String _
                                               , ByVal RegionName As String _
                                               , ByVal EmulatorSessionID As String _
                                               , ByVal DataTruncationFlag As String _
                                               , ByVal SessionToken As String _
                                               , ByVal StandardWait As String _
                                               , ByVal ApplicationVersion As String _
                                               , ByVal Environment As String _
                                               , ByVal ESILogonOnly As String _
                                               , ByVal ScreenDescDataSource As String 
                                               , ByRef ScreenDataCacheCount As Short) 

When I called this method, I submitted null terminated char arrays ex.

Dim param1 As Char()
param1(0) = "c"c
param1(1) = "a"c
param1(2) = "t"c
param1(3) = ChrW(0)
StartSession (param1,...)

The pinvoke wizard generated the following stub

<DllImport("chat2.dll")> _
Public Function StartSession( _
        ByRef lpbReturnCode As Byte _
        , ByRef lpbErrorMessage As Byte _
        , ByRef lpbApplName As Byte _
        , ByRef lpbRegionName As Byte _
        , ByRef lpbSessionID As Byte _
        , ByRef lpbDataTrunc As Byte _
        , ByRef lpbSessionToken As Byte _
        , ByRef lpbStandardWait As Byte _
        , ByRef lpbVersion As Byte _
        , ByRef lpbEnvironment As Byte _
        , ByRef lpbESILogonOnly As Byte _
        , ByRef lpbMetadataSource As Byte _
        , ByRef lpbMetadataCacheCount As Short _
        ) As Short
End Function

How can I use a single byte to represent a null terminated string of characters?

Answer 1

To the question:

How can I use a single byte to represent a null terminated string of characters?

The simple answer is "You don't."

That's not really what is going on in the code anyway. First off, ByRef someVar As Byte is not passing a Byte to the API. It is passing a pointer to a byte. So on the native side, the code is getting an address of that byte. Now if that address points the the first byte in a buffer of bytes containing your string's characters and is (likely) null-terminated, it is easy for that API to read that content upto that null and act upon it.

And you could actually use unsafe code blocks in C# or maybe GCHandle pinng or something in the Marshal class or MarshalAs attributes to mimic this by doing something like StartSession(myParam1Buff[0], etc, etc);.

But don't. You need to read the docos (not just the header files) and determine what the API is expecting and the use StringBuilder or 'String instances decorated with the correct MarshalAs attribute with the UnmanagedType flag set based on the doco. Most likely, it will be LPStr.