CODEWITHSUNDEEP
c#floating-pointnumber-formatting
Bernhard Hiller
Bernhard Hiller

Reputation: 2397

C# Float formatting quirks

Attention: This is not a question about lack of precision in floats when doing mathematical operations or comparisons. It is about formatting issues, though the confusion caused may leak in the description.

Background: A log entry of our application essentially said that 19 was greater than 19.

Reproduction: The mathematical steps leading to the issue could be found out, and the problem can be reproduced in a simple test:

[TestMethod]
public void WtfTest()
{
    float bottom = -51;
    float top = 19;
    float rowHeight = (float)((top - bottom) / 3.0);
    float line1 = bottom + rowHeight;
    float line2 = line1 + rowHeight;
    float line3 = line2 + rowHeight;
    Assert.IsFalse(line3 > top, $"WTF: line3={line3:n50} is greater than top={top:n50}.");
}

This test fails because line3 is actually greater than top, as can be checked e.g. with BitConverter.GetBytes. That's NOT the issue here. The real issue is the failure message which perfectly reproduces the original problem:

Assert.IsFalse failed. WTF: line3=19,00000000000000000000000000000000000000000000000000 is greater than top=19,00000000000000000000000000000000000000000000000000.

You see: the two numbers shown are perfectly equal. Though they are actually different.

Additional findings: Calculating the difference and formatting it yields 0,00000190734900000000000000000000000000000000000000 which seems correct.

Debugging the test and looking at the values in VisualStudio's "Locals" window shows 19.0000019 for line3.

Environment: .Net Framework 4.8, platform target x64, on Windows 10

Final question: How can I get the number formatting right?

Upvotes: 2

Views: 97

Answers (1)

shingo
shingo

Reputation: 27046

According to the documentation, the recommended format for float is "G9" and for double is "G17".

When used with a Double value, the "G17" format specifier ensures that the original Double value successfully round-trips. This is because Double is an IEEE 754-2008-compliant double-precision (binary64) floating-point number that gives up to 17 significant digits of precision.

When used with a Single value, the "G9" format specifier ensures that the original Single value successfully round-trips. This is because Single is an IEEE 754-2008-compliant single-precision (binary32) floating-point number that gives up to nine significant digits of precision.

I've tested the following code on .net framework and .net core

float bottom = -51;
float top = 19;
float rowHeight = (float)((top - bottom) / 3.0);
float line1 = bottom + rowHeight;
float line2 = line1 + rowHeight;
float line3 = line2 + rowHeight;
Console.WriteLine(line3.ToString("G9"));
Console.WriteLine(line3.ToString("N50"));

Here is the result

//.net framework
19.0000019
19.00000000000000000000000000000000000000000000000000

//.net core
19.0000019
19.00000190734863281250000000000000000000000000000000

Upvotes: 3

Related Questions