CODEWITHSUNDEEP
assemblycalldinline-assembly
Moritz Schöfl
Moritz Schöfl

Reputation: 763

D Inline Assembler: error with function call

I got a very special problem. For a VM I need to copy code from the instruction functions to a ubyte array and then execute this array (the technic is similiar to the inline macro vm in gcc), basically it works like this:

__gshared void * sp = null, sb = null; //stack pointer and stack base

__gshared void add() //the function is just there to access the instruction code
{
    asm{db "INSTRUCTIONCODESTART";} //this is a key to know where the instruction code starts

    //instruction code here (sample instruction add, pops 2 values from the stack and pushes its result)
    sp += 4;
    *cast(uint*)sp += *cast(uint*)(sp - 4);

    asm{db "INSTRUCTIONCODEEND";} //this is a key to know where instruction code ends
}

In the Init method, every instruction code gets its own buffer, and in the buffer is every byte between the INSTRUCTIONCODESTART and the INSTRUCTIONCODEEND key. I make this array executeable through the windows VirtualProtect call.

So Far, everything works as expected, but when I try to do a function call as an instruction, I will get an error.

__gshared void testcall(){}

__gshared void call()
{
    asm{db "INSTRUCTIONCODESTART";} //this is a key to know where the instruction code starts

    //instruction code here (just calls a D function)
    testcall(); //this somehow throws an error

    asm{db "INSTRUCTIONCODEEND";} //this is a key to know where instruction code ends
}

Btw I tested the instructions with the following code

void instructiontest()
{
    uint dummy;
    ubyte[] buf = getFunctionCode(&add) ~ 0xC3; //gets code of instruction, appends 0xC3 at it ("ret" instruction, for test purposes only to see if it returns to the D code without errors)
    VirtualProtect(cast(void*)buf, buf.length, PAGE_EXECUTE_READWRITE, &dummy); //makes it executeable
    dummy = cast(uint)&buf[0];
    asm
    {
        call dummy[EBP];
    }
    print("instruction worked without errors!");
}

So far, every simple instruction (add, mul, sub, push0, push1, ...) works, but if I try to get the code of an instruction with a function call, it throws an error

I would be happy and very thankful about any help. (btw I need function calls in the instruction in order to let the script language communicate with D)

Upvotes: 1

Views: 412

Answers (2)

user438034
user438034

Reputation:

You should really disassemble the code in order to get a clear view of what it's doing and why your code is breaking. The disassembly for your call function is:

0000000000414db8 <_D4test4callFZv>:
  414db8:   55                      push   rbp
  414db9:   48 8b ec                mov    rbp,rsp
  414dbc:   48 83 ec 08             sub    rsp,0x8
  414dc0:   53                      push   rbx
  414dc1:   41 54                   push   r12
  414dc3:   41 55                   push   r13
  414dc5:   41 56                   push   r14
  414dc7:   41 57                   push   r15
  414dc9:   49                      rex.WB
  414dca:   4e 53                   rex.WRX push rbx
  414dcc:   54                      push   rsp
  414dcd:   52                      push   rdx
  414dce:   55                      push   rbp
  414dcf:   43 54                   rex.XB push r12
  414dd1:   49                      rex.WB
  414dd2:   4f                      rex.WRXB
  414dd3:   4e                      rex.WRX
  414dd4:   43                      rex.XB
  414dd5:   4f                      rex.WRXB
  414dd6:   44                      rex.R
  414dd7:   45 53                   rex.RB push r11
  414dd9:   54                      push   rsp
  414dda:   41 52                   push   r10
  414ddc:   54                      push   rsp
  414ddd:   e8 ce ff ff ff          call   414db0 <_D4test8testcallFZv>
  414de2:   49                      rex.WB
  414de3:   4e 53                   rex.WRX push rbx
  414de5:   54                      push   rsp
  414de6:   52                      push   rdx
  414de7:   55                      push   rbp
  414de8:   43 54                   rex.XB push r12
  414dea:   49                      rex.WB
  414deb:   4f                      rex.WRXB
  414dec:   4e                      rex.WRX
  414ded:   43                      rex.XB
  414dee:   4f                      rex.WRXB
  414def:   44                      rex.R
  414df0:   45                      rex.RB
  414df1:   45                      rex.RB
  414df2:   4e                      rex.WRX
  414df3:   44                      rex.R
  414df4:   41 5f                   pop    r15
  414df6:   41 5e                   pop    r14
  414df8:   41 5d                   pop    r13
  414dfa:   41 5c                   pop    r12
  414dfc:   5b                      pop    rbx
  414dfd:   c9                      leave  
  414dfe:   c3                      ret    
  414dff:   90                      nop

414dc9 is where the start marker begins, 414ddc is where it ends (inclusive). 414de2 is where your end marker begins, 414df3 is where it ends (inclusive). So, ripping that out, we have:

0000000000414db8 <_D4test4callFZv>:
  414db8:   55                      push   rbp
  414db9:   48 8b ec                mov    rbp,rsp
  414dbc:   48 83 ec 08             sub    rsp,0x8
  414dc0:   53                      push   rbx
  414dc1:   41 54                   push   r12
  414dc3:   41 55                   push   r13
  414dc5:   41 56                   push   r14
  414dc7:   41 57                   push   r15
  ; code start marker here
  414ddd:   e8 ce ff ff ff          call   414db0 <_D4test8testcallFZv>
  ; code end marker here
  414df4:   41 5f                   pop    r15
  414df6:   41 5e                   pop    r14
  414df8:   41 5d                   pop    r13
  414dfa:   41 5c                   pop    r12
  414dfc:   5b                      pop    rbx
  414dfd:   c9                      leave  
  414dfe:   c3                      ret    
  414dff:   90                      nop

You're clearly not copying some prologue and epilogue code here. But that, in and of itself, should not be terribly problematic.

I tried this program:

void main()
{
    foo();
}

void foo()
{
    auto addr = &bar;

    asm { call addr; }
}

void bar()
{
    asm { naked; call baz; ret; }
}

void baz()
{
}

It works for me. Frankly, I can't tell where your problem is. Most of the code you pasted can't just be copied into a source file and be compiled, so it's quite hard to tell what's going wrong. I hope some of the information here can help you. You're most likely going to have to attach a debugger and find out what's wrong; don't expect to mess with low-level stuff like this without getting your hands dirty. ;)

BTW, I tested on 64-bit x86 in Linux.

Either way, what you're doing is highly unportable, undefined, etc. Be warned. It may work out, but you're working with zero guarantees.

Upvotes: 2

Alexey Frunze
Alexey Frunze

Reputation: 62106

It's hard to tell what all the mistakes are without seeing the full disassembly of all the relevant code, but...

  1. x86 code generally isn't position-independent, which means copying it to a different place and executing it there may and most often will fail.

  2. You're most likely copying code that trashes registers (including ebp and esp) and the stack contents.

Upvotes: 0

Related Questions