CODEWITHSUNDEEP
assemblyx86masminteger-overflowstack-pointer
jww
jww

Reputation: 102356

Prologue of "add esp, 0FFFFFFF8h"

I'm a tad bit rusty on my MASM, so I don't really recall what to do here (if anything needs to be done at all). I have a MASM (X86) routine that looks as follows. It has two local variables taking up 5 bytes total:

MSC_ASM_GenerateBlock PROC buffer:DWORD,bsize:DWORD,safety:DWORD
  LOCAL val:DWORD, rc:BYTE  ;; local variables
  MWSIZE EQU 4              ;; machine word size

  .WHILE bsize >= MWSIZE && safety > 0
     ;; RDRAND is not available prior to VS2012. Just emit
     ;;   the byte codes using DB. This is `rdrand eax`.
     DB 0Fh, 0C7h, 0F0h
     setc rc
     ...
  .ENDW
  ...
MSC_ASM_GenerateBlock ENDP

When I check the disassembly, I see:

> dumpbin.exe /DISASM rdrand-x86.obj
Dump of file rdrand-x86.obj

_MSC_ASM_GenerateBlock:
  00000000: 55                 push        ebp
  00000001: 8B EC              mov         ebp,esp
  00000003: 83 C4 F8           add         esp,0FFFFFFF8h
  00000006: EB 1D              jmp         00000025
  00000008: 0F C7 F0           rdrand      eax
  0000000B: 0F 92 45 FB        setb        byte ptr [ebp-5]
  0000000F: 80 7D FB 00        cmp         byte ptr [ebp-5],0
  ...

I believe add esp, 0FFFFFFF8h is another way to say sub esp, 08h.

As Joshua pointed out, the difference between an add esp and sub esp is the flags after the operation. The assembler's confusion or selection of instructions could be based on the fact that the assembler does not get to see the RDRAND context. Rather, it only sees the jmp based on CY, and the assembler believes the flags are not in a good state.

Why is MASM generating a non-intuitive add that depends on unsigned integer wrap? And more importantly, is it OK?

I performed a port of the routing to MASM64/ML64 using the wider machine words. It produces the same code (modulo machine word size):

Dump of file rdrand-x64.obj

MSC_ASM_GenerateBlock:
  0000000000000000: 55                 push        rbp
  0000000000000001: 48 8B EC           mov         rbp,rsp
  0000000000000004: 48 83 C4 F0        add         rsp,0FFFFFFFFFFFFFFF0h
  0000000000000008: EB 1D              jmp         0000000000000037
  0000000000000010: 48 0F C7 F0        rdrand      rax
  000000000000001D: 0F 92 45 F7        setb        byte ptr [rbp-9]
  0000000000000024: 80 7D F7 00        cmp         byte ptr [rbp-9],0
  ...

Upvotes: 3

Views: 1992

Answers (2)

dave
dave

Reputation: 4922

I have no idea why MASM would think that's a good idea but unsigned overflow will work (so this is safe), and even more confusingly this is a safe substitution to make even on a 64 bit processor because the 32 bit operand will zero out the high 32 bits of the register! See this for an explanation.

Also, since both instructions are 3 bytes long there isn't any inefficiency there.

Upvotes: 0

Joshua
Joshua

Reputation: 43317

Odd compiler. Very odd.

add esp, 0FFFFFFF8h

is exactly the same as

sub esp, 8h

except it sets the flag bits differently. It's fine, and yes it depends on unsigned integer wrap. Not a problem because assembly is inherently non-portable. If you wanna know why you'll have to ask Microsoft, and they probably don't know anymore.

Upvotes: 7

Related Questions