How to translate DLL Function Call / DllImport from vb6 to C# .NET

Question

The code in VB6 works perfectly fine, I believe the dll is coded in assmebly for speed, I'm trying to port calling it in C# or VB.NET whichever I don't mind.. I tried many different tricks nothing works, It should be called similar to how RltMoveMemory (CopyMemory) is called with a Array address passed by offset.

The VB6 code looks like this

Public Type REGTYPE
  REG_Kind As Byte  ' ;1=8 bits \ 2=16 bits \ 3=32 bits \ 4=MMX \ 5=XMM \ 6=Float stack \ 7=Segment \ 8=Debug \ 9=Control \ 10=Test
  REG_Ptr_Kind As Byte  ' ;1=Byte PTR \ 2=Word PTR \ 3=Dword PTR \ 4=Qword PTR \ 5=mmword ptr \ 6=xmmword ptr \ 7=FWord PTR \ 8=tbyte ptr \ 9=null ptr (LEA)
  REG_Type As Byte '  ;0-7= direct register index \ 16 register=byte && 7 \ 32 register=(byte && 63)/8 \ 64=[32/16 address only] \ 128=[using x86 relatives]
  REG_BaseAsReg As Byte ' ? ;1=Register only (BASE exposed)!
End Type

Public Type REGSTRUCT
 SEG_TYPE As Long
 Base As Long
 INDEX As Long
 SCALE As Long
 DISPLACEMENTS As Long
 DISPLACEMENT_TYPE As Long
 REG_Kind As REGTYPE
 PTR_TYPE As Long
End Type

Public Type IMMSTRUCT
 VALUE_LO As Long
 VALUE_HI As Long
 VALUE_TYPE As Long '     1=Byte \ 2=Word \ 4=Dword \ 8=ByteToWord \ 16=ByteToDword \ 32=AbsJump \ 64=ShortJump \ 128=LongJump
End Type

Public Type DisAsmStruct
 Instruction_Prefix As Long
 Instruction As Long
 Reg1 As REGSTRUCT
 Reg2 As REGSTRUCT
 Reg_Reg As Long '1=from ptr
 Imm As IMMSTRUCT
 Instruction_Length As Long
End Type

'return buffer length
Declare Function DisAssemble Lib "disASM" (Data As Any, ByVal BaseAddress As Long, DisAsmString As Any, DisAsmS As Any, ByVal DisasmOpt As Long) As Long

Say you load file as binary to, the struct is initialized like so

Dim FDATA() as byte
Dim DisA As DisAsmStruct

The DLL Call would be

BufferLength = DisAssemble(FDATA(CNT), BaseAddress + CNT, ByVal Opcodes, DisA, 0)

The CNT value is a counter which gets incremented after the call based on how long the disassembled instruction was found to be

CNT = CNT + DisA.Instruction_Length

The other values in the DLL Function OpCodes is written string memory where the string of the human-readable asm code is printed. Also DisA struct is filled out by that same DLL Function call. The BufferLength is returned as a function return value I don't think I'd have any problems with that one it's used to trim the OpCodes string to proper string size later.

The OpCodes is declared like so.

Dim Opcodes     As String
Opcodes = String(128, 0)

Here is what I tried so far in C#

All my failure attempts

    //return buffer length
    [DllImport("disASM.dll", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern int DisAssemble(IntPtr Data, uint BaseAddress, IntPtr DisAsmString, IntPtr DisAsmS, uint DisasmOpt);

    [DllImport("disASM.dll", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern int DisAssemble(IntPtr Data, uint BaseAddress, string DisAsmString, DisAsmStruct DisAsmS, uint DisasmOpt);

    [DllImport("disASM.dll", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern short DisAssemble(
      [MarshalAs(UnmanagedType.LPArray)] [In] ref byte[] data, 
      uint BaseAddress,
      ref string DisAsmString,
      ref DisAsmStruct DisAsmS,
      uint DisasmOpt);

    [DllImport("disASM.dll", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern short DisAssemble(
      [MarshalAs(UnmanagedType.LPArray)] [In] ref byte[] data, 
      uint BaseAddress,
      ref string DisAsmString,
      ref DisAsmStruct DisAsmS,
      uint DisasmOpt);

    [DllImport("disASM.dll", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern short DisAssemble(
      [MarshalAs(UnmanagedType.LPArray)] [In] ref byte[] data, 
      uint BaseAddress,
      [MarshalAs(UnmanagedType.LPStr)]  ref string DisAsmString,
      [MarshalAs(UnmanagedType.LPStruct)]ref DisAsmStruct DisAsmS,
      uint DisasmOpt);

//this looks like the best one
        [DllImport("disASM.dll", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
        public static extern int DisAssemble(IntPtr Data, uint BaseAddress, out string DisAsmString, out DisAsmStruct DisAsmS, uint DisasmOpt);

I cannot do like

    [DllImport("disASM.dll", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern short DisAssemble(
      byte* data, 
      uint BaseAddress,
      string* DisAsmString,
      DisAsmStruct* DisAsmS,
      uint DisasmOpt);

because it would give error Pointers and fixed size buffers may only be used in an unsafe context

The closest attempt was this, it didn't work but I just feel it was good enough attempt as the pointer is exposed and incremented properly with CNT counter, just it's not getting passed due to some protection in .NET i don't know.

unsafe
{ 
    fixed(byte* test = &GLOBALDATA.FDATA[CNT])
    {
        IntPtr codeBuf = new IntPtr((void*)test);
        BufferLength = ModuleASM.DisAssemble(codeBuf, BaseAddress + CNT, out Opcodes, out DisA, 0);
    }
}

Online research said I do like this (doesn't work of course)

GCHandle fileDataPointered = GCHandle.Alloc(GLOBALDATA.FDATA[CNT], GCHandleType.Pinned);
BufferLength = ModuleASM.DisAssemble(fileDataPointered.AddrOfPinnedObject(), BaseAddress + CNT, ref Opcodes, ref DisA, 0);
//BufferLength = ModuleASM.DisAssemble(fileDataPointered.AddrOfPinnedObject(), BaseAddress + CNT, Opcodes, DisA, 0);
fileDataPointered.Free();

Online research example compiles fine but similar error to the unsafe code example

An unhandled exception of type 'System.AccessViolationException' occurred in {$1}.exe
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

There was one tool that someone recommanded on google link I try called P/Invoke Interop Assistant which is suppose to generate a DLLImport based on loading a .DLL file into it.. I get a error from that as well that the .dll file is missing a assembly manifest.

Answer 1

Yay! I got it to call.

The problem was that .NET string is different from vb6 String I don't know the difference exactly.. but vb6

Dim OpCodes As String

changing C#

string OpCodes;

to

IntPtr OpCodes;

completely fixed the problem, Yes it's not pretty but I have to take it out of the pointer back to string now, i'll work on that now.

At first I thought maybe I had to move all the variables into the unsafe scope.. I did that, no idea if it contributed to the fix.. but I think not that important to do, yes I tested the variables outside the unsafe scope still works fine!.

Full working code looks like this, okay it's not working for strings.. I'll keep at it, fixed that too

[DllImport("disASM.dll", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
public static extern int DisAssemble(IntPtr Data, uint BaseAddress, out IntPtr DisAsmString, out DisAsmStruct DisAsmS, uint DisasmOpt);


unsafe
{
    string Opcodes = new string((char)0, 128);
    IntPtr OpcodesTest;
    ModuleASM.DisAsmStruct DisAa = new ModuleASM.DisAsmStruct();
    fixed (byte* dataPtr = &GLOBALDATA.FDATA[CNT])
    {
        IntPtr dataBuf = new IntPtr((void*)dataPtr);
        BufferLength = ModuleASM.DisAssemble(dataBuf, BaseAddress + CNT, out OpcodesTest, out DisAa, 0);

        //Kinda like it.. need more characters like, PUSH ECX
        ASCIIEncoding.ASCII.GetString(BitConverter.GetBytes(OpcodesPtr.ToInt64())) //return "PUSH\0\0\0\0"

        byte testbbb = Marshal.ReadByte(OpcodesTest); //fail error
        string testa = Marshal.PtrToStringAnsi(Marshal.ReadIntPtr(OpcodesTest)); //fail error
        string testb = Marshal.PtrToStringAnsi(OpcodesTest); //blank return
        string testc = Marshal.PtrToStringUni(OpcodesTest); //fail error
        string testd = Marshal.PtrToStringUni(Marshal.ReadIntPtr(OpcodesTest)); //fail error
        string teste = Marshal.PtrToStringAuto(Marshal.ReadIntPtr(OpcodesTest)); //fail error
        string testf = Marshal.PtrToStringAuto(OpcodesTest); //fail error
    }
}

Answer 2

You could be having issues with the platform with 32 bit vs 64 bit. VB6 is 32 bit only so DisAssemble.dll is a 32bit dll.

.Net supports both 32 bit and 64 bit and defaults to AnyCpu. AnyCpu will run as 64 bit on a 64 bit operating system, which is most likely the case.

Try change the Target Platform to x86.

You cannot load a 32bit dll in a 64 bit process. Also IntPtr is 8 bytes when running on a 64 bit CLR and 4 bytes when running on the 32 bit CLR.