CODEWITHSUNDEEP
vb.netcomparisonlogical-operators
Justin8051
Justin8051

Reputation: 429

Why is this basic number comparison producing an illogical result?

While hunting a weird bug in my VB.NET application, I tracked it down to a shockingly puzzling detail. Here is a simple test code:

If 0.01 > 0.12 - 0.11 Then Debug.Print("what the hell")

0.12-0.11 is 0.01... Which is equal to the left side of the comparison. However, when I run this, the debug prints "what the hell"... Because seriously, what the hell. These numbers are equal.

Additionally, if I have a cycle like this:

Dim count As Integer = 0
For i As Double = 0.11 to 0.12 Step 0.01
   count += 1
Next
Debug.Print(count)

It prints 1, meaning the cycle is executed only once, while it should execute twice.

Surprisingly, if I change 0.11, 0.12 and 0.01 in the above examples to 0.1, 0.2 and 0.1, then first example doesn't print anything, and the second example prints 2, as it should.

What is going on here? Am I missing something incredibly obvious, or is this some kind of floating point error or something?

Upvotes: 1

Views: 101

Answers (3)

Chris Dunaway
Chris Dunaway

Reputation: 11216

If you can't use Decimal for your calculations, then you must write your code to account for the fact that binary floating point types can't represent some fractional values exactly. So rather than checking if the numbers are equal, you check if they are nearly equals.

You can use code like the following (taken from this article by Michael Borgwardt:

This is the VB translation, but not tested extensively. 

Public Shared Function NearlyEqual(a As Double, b As Double, epsilon As Double) As Boolean
    Dim absA As Double = Math.Abs(a)
    Dim absB As Double = Math.Abs(b)
    Dim diff As Double = Math.Abs(a - b)

    If (a = b) Then
        'shortcut, handles infinities
        Return True
    Else 
        If (a = 0 OrElse b = 0 OrElse diff < Double.Epsilon) 
            'a or b is zero or both are extremely close to it
            'relative error is less meaningful here
            Return diff < epsilon
        Else
            'use relative error
            Return diff / (absA + absB) < epsilon
        End If
    End If
End Function

Upvotes: 0

Mary
Mary

Reputation: 15091

How about Integer arithmetic?

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
    Dim count As Integer = 0
    For i As Integer = CInt(0.11 * 100) To CInt(0.12 * 100) Step CInt(0.01 * 100)
        count += 1
    Next
    Debug.Print(count.ToString)
End Sub

Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
    If CInt(0.01 * 100) > CInt(0.12 * 100) - CInt(0.11 * 100) Then
        Debug.Print("what the hell")
    Else
        Debug.Print("It's Ok")
    End If
End Sub

Upvotes: 0

R.J. Dunnill
R.J. Dunnill

Reputation: 2089

You're getting these problems because you're using floating-point types, which use base 2, and base 2 can't represent some fractional values exactly.

That's why fixed-point types like Decimal were devised. If your example code is reworked for fixed-point (using Decimal), it gives the expected results.

    ' Specify Decimal constants and this will worked as anticipated.
    If 0.01D > 0.12D - 0.11D Then Debug.Print("what the hell")

    ' 0.12-0.11 Is 0.01... Which Is equal to the left side of the comparison.
    ' However, when I run this, the debug prints "what the hell"... Because 
    ' seriously, what the hell. These numbers are equal.

    ' Additionally, If I have a cycle Like this

    Dim count As Integer = 0
    For i As Decimal = 0.11D To 0.12D Step 0.01D
        count += 1
    Next

    Debug.Print(count) ' Prints 2

Upvotes: 1

Related Questions