Confused about assembly FLD instruction m64fp

Question

I'm so confused. I have some question about the FLD m64fp instruction, but I have no idea where to start. Because this is a homework, I'm not specifically asking for answers, but the method to solve the problem. Any suggestion or idea would be appreciated.

Eight consecutive bytes in memory contain the hex values 01, 00, 00, 00, 00, 00, 00, 00. An FLD m64fp instruction is executed. Its argument is the address of the first of these eight consecutive bytes. As a result of the FLD instruction, the value in ST(0) is now:

1)  2^(-1075)
2)  2^(-1074)   
3)  2^(-1023)
4)  2^(-1022) 

Also, if I have this following assembly code

   │0x8048384 <main>                lea    0x4(%esp),%ecx                                                                                                                                               │
   │0x8048388 <main+4>              and    $0xfffffff0,%esp                                                                                                                                             │
   │0x804838b <main+7>              pushl  -0x4(%ecx)                                                                                                                                                   │
   │0x804838e <main+10>             push   %ebp                                                                                                                                                         │
   │0x804838f <main+11>             mov    %esp,%ebp                                                                                                                                                    │
   │0x8048391 <main+13>             push   %ecx                                                                                                                                                         │
   │0x8048392 <main+14>             fldpi                                                                                                                                                               │
   │0x8048394 <main+16>             fsqrt                                                                                                                                                               │
   │0x8048396 <main+18>             fld1                                                                                                                                                                │
   │0x8048398 <main+20>             fsubrp %st,%st(1)                                                                                                                                                   │
   │0x804839a <main+22>             mov    $0x0,%eax                                                                                                                                                    │
   │0x804839f <main+27>             pop    %ecx                                                                                                                                                         │
   │0x80483a0 <main+28>             pop    %ebp                                                                                                                                                         │
   │0x80483a1 <main+29>             lea    -0x4(%ecx),%esp                                                                                                                                              │
   │0x80483a4 <main+32>             ret   

How do I find out the value that will be in ST(0) just before main returns? Thank you.

Answer 1

Let's start with the first question.

It isn't explicitly stated, but I think we can assume we're dealing with little-endian here (every PC you'll get to use today will use that). Thus, if you execute FLD m64p on that memory location, the floating point stack will contain these bytes in reversed order - i.e. 00 00 00 00 00 00 00 01. Let's look what the double precision format looks like:

Now, this is actually a special case - since the exponent is zero and the mantissa isn't, we're representing a subnormal number - one that cannot be expressed using a normalized mantissa, i.e. where the integer part is 1 (1.xxx) - it needs leading zeros (remember, since the exponent is biased (1023), zero actually means 1 - exponent (0) - 1023 here, so -1022.

As Wikipedia tells us, we can calculate the value of a subnormal number using the following formula:

However, the least significant bit (and only that one) in the mantissa is set, and this gives the mantissa a value of 2^(-52) (since we have 52 bits for the mantissa in the double precision format).

Therefore, if we use that formula, what we get is:
(-1)^0 x 2^(1-1023) x 2^(-52) = 1 x 2^(-1022) x 2^(-52) = 2^(-1022 - 52) = 2^(-1074), which is answer 2.

This is the smallest possible positive subnormal number - if that last bit were not set, the bits would represent a signed zero.

To test that (or to find out the result more easily, if you're feeling lazy :)) you could use OllyDbg for Windows, which allows you to modify assembly code on the fly. Let's enter the instruction given in the question:

and let's set the bytes to the required value:

Indeed, when we execute that, we get this:

Which is (almost) equal to 2 ^ (-1074).

Now, about the second question. Let's analyze the instructions you have listed.

1. We start with fldpi, which is equivalent to ST(0) = PI.
2. We execute fsqrt, so now we have ST(0) = sqrt(PI).
3. fld1 loads one into ST(0), so the stack looks like this: ST(0) = 1, ST(1) = sqrt(PI).
4. fsubrp performs reversed subtraction and pops the register stack. Since this is AT&T assembly, and a buggy one at that, the source is first, so we subtract ST(0) from ST(1), store the result in ST(1), and pop the register stack so that ST(1) becomes ST(0). Effectively, now we have ST(0) = sqrt(PI) - 1, which is close to 0.772. This value remains in ST(0) when main returns since nothing modifies the floating point stack later.