I do not understand what Machine Epsilon means

Question

I'm studying Matlab and computer arithmetic, but I'm having trouble grasping the concept of the Machine Epsilon. I would appreciate a simple explanation, as if I were a child. What does '2.2204e-16' mean, what are the issues when we add '1' and what is its purpose?

I've only thought that with a 64-bit machine epsilon, we know that a number is accurate up to the sixteenth digit, but I have many doubts.

Answer 1

Non-zero finite floating-point numbers are represented as ±f•b^e, where:

• b is a fixed integer that is the base of the format (most commonly 2),
• f is a numeral in base b with p digits, and
• e is a integer exponent in a certain interval, e_min ≤ e ≤ e_max.

f is called the significand. p is the precision of the format. f is restricted to be in a certain interval. For the moment,¹ we take 1 ≤ f < b. For example, with b = 2 and p = 53 (the precision for the format commonly used for double), the number 1 is represented as:

• +1.0000000000000000000000000000000000000000000000000000₂•2⁰.

That is “1.” followed by 52 “0” digits, so it is a 53-bit numeral in base 2.

The so-called machine epsilon is the smallest change we can make to get to the next representable number. That next number is:

• +1.0000000000000000000000000000000000000000000000000001₂•2⁰.

So the difference is:

• • +0.0000000000000000000000000000000000000000000000000001₂•2⁰.

That is 2⁻⁵²•2⁰ = 2⁻⁵² ≈ 2.220446049•10⁻¹⁶.

This is the Unit of Least Precision (ULP) of 1.

Note that the ULP depends on the general size of the number, specifically on which exponent is used to represent it. The number 32,768 is represented with an exponent of 15:

• +1.0000000000000000000000000000000000000000000000000000₂•2¹⁵.

So the next representable number of that is:

• +1.0000000000000000000000000000000000000000000000000001₂•2¹⁵.

which means the difference is:

• +0.0000000000000000000000000000000000000000000000000001₂•2¹⁵.

which is 2⁻⁵²•2¹⁵ ≈ 7.275957614•10⁻¹².

So the ULP of 32,768 is about 7.275957614•10⁻¹².

Supplement

I've only thought that with a 64-bit machine epsilon, we know that a number is accurate up to the sixteenth digit, but I have many doubts.

The number of bits in a format can only tell you the precision of numbers in the format, not how accurate they are. When you convert a number to a floating-point format, it is rounded to the format. When you add, multiply, or perform other operations, the ordinary arithmetic result is rounded to the format. Over the course of various operations, these rounding errors may compound or cancel, so a final result may be very inaccurate or may be accurate. Determining which and by how much can be difficult. There are some sequences of operations that almost always produce inaccurate results no matter how precise a floating-point format is. Precision does not tell you about accuracy.

Footnote

¹ Sometimes, f is scaled to the interval [b ^p−1, b ^p[, and the exponent is adjusted accordingly. This makes f always an integer, so it is useful for using number theory in proofs about floating-point. Sometimes, it is scaled to [1/b, 1[. The C standard uses this. In all of those, the first digit of f must be non-zero. Sometimes, the first digit is allowed to be zero, which allows representing zero and subnormal numbers.

Answer 2

What does '2.2204e-16' mean, what are the issues when we add '1' and what is its purpose?

Using typical 64-bit floating point, epsilon is about 0.00000000000000022204 ('2.2204e-16' or 2.2204*10^-16) or exactly 2^-52.

Epsilon: the difference between 1 and the least normalized value greater than 1 that is representable in the given floating type.

...
Between [0.5 and 1.0], 64-bit floating point steps are each 2^-53.
Between [1.0 and 2.0], 64-bit floating point steps are each 2^-52. Epsilon
Between [2.0 and 4.0], 64-bit floating point steps are each 2^-51.
Between [4.0 and 8.0], 64-bit floating point steps are each 2^-50.
...

---

we know that a number is accurate up to the sixteenth digit, but I have many doubts.

Typical floating point, if written in base 2: ±b.bbbbbbbb... * 2^exponent is precise to 53 binary digits.
That is about 15.95 or log₁₀(2⁵³) decimal digits ±d.ddddddd... * 10^exponent.