CODEWITHSUNDEEP
assemblyx86ssedisassemblyollydbg
abortedReverseEngineer
abortedReverseEngineer

Reputation: 1

OllyDbg not properly interpreting SSE2 instructions and operands

Note: using OllyDbg v1.1

I'm attempting to reverse engineer a subroutine in a program that likely employs some sort of techniques to frustrate the process beyond the basic 'IsDebuggerPresent'.

I have the code disassembled in IDA, but certain instructions aren't being identified as any valid instruction by OllyDbg. This occurs a number of times, and often causes later instructions to mess up their alignment. Note that the code executes, I just can't set breakpoints in the region I need to in order to look at a particular value on the stack before a specific function call.

In particular, the instruction which IDA disassembles as SSE2 movq [ebp+var_DF4], xmm0 seems to be problematic (see below).

Given var_DF4 = -0x0DF4, I think Olly should be identifying it as
MOVQ QWORD PTR [EBP-0xDF4],XMM0 like IDA does.

I tried to force Olly to do this by manually attempting to assemble the instruction, but I got the error 'Command Does Not Support Given Operands.'

Here is the successfully disassembled code seen in IDA:
Here is the successfully disassembled code seen in IDA

Here is what Olly shows:
Here is what Olly shows

Upvotes: 0

Views: 213

Answers (2)

Alex X.
Alex X.

Reputation: 301

OllyDbg not properly interpreting SSE2 instructions and operands that's right! but

You could try the super mature and professional tool OllyDbg version 2.01 https://www.ollydbg.de/version2.html

It knows very well how to take care of XMM/YMM registers as well as its Disassembler supports integer, FPU, MMX, 3DNow, SSE1-SSE4.1 and AVX operands

Upvotes: 1

Peter Cordes
Peter Cordes

Reputation: 364532

movq is an SSE2 instruction (https://www.felixcloutier.com/x86/movd:movq). xorps and movups are SSE1 instructions, so it seems your version of Ollydbg only knows about SSE1.

That would explain everything you're seeing, and the only solution is to use newer software that knows how to disassemble the instruction set extensions your code uses.

From the disassembly, it's pretty clear that Ollydbg just gives up at the 2-byte 0F D6 opcode with a 66 prefix. It treats that as the end of an instruction it doesn't know. (66 is the operand-size prefix, and is a "mandatory prefix" as part of the encoding for movq.)

It doesn't assume the unknown instruction has a ModRM byte, so it starts decoding again at the 85 byte, which was supposed to be the ModRM for your movq, but if you start decoding there it's the opcode for TEST r/m32, r32, so that's what Olly sees. The displacement bytes get decoded as ModRM and SIB for that test, leaving two displacement bytes (FF FF) left, which get decoded as another unknown instruction.

Upvotes: 3

Related Questions