CODEWITHSUNDEEP
vbaexcel
Glenn G
Glenn G

Reputation: 667

Unexpected Rounding Error

can someone please explain the following to me?

Sub TestCalc()
    Dim Z As Double
    Dim Y As Double
    Dim X As Integer
    Dim W As Double
    Dim V As Double
    X = 44 / 14                         ' returns 3
    Z = (0.14 * 14)                     ' returns 1.96
    Y = ((44 / 14) - (44 \ 14)) * 14    ' returns 2 SHOULD RETURN 1.96
    W = (44 / 14) - X                   ' returns 0.142857142857143
    V = W * 14                          ' returns 2 SHOULD RETURN 1.96
End Sub

1.96 is the value that I would expect to get from the code. However, I only get this value when I use hard coded values. If I work with variables it rounds it up and returns the value 2 (Y or V). I'm need to understand why, as 1.96 is that value that I expect to be returned. I need to ensure that it performs this calculation correctly to ensure that my math formula functions properly in my main procedure

Upvotes: 0

Views: 58

Answers (2)

PKatona
PKatona

Reputation: 639

Your expectations are incorrect.

0.14 * 14 = 1.96; however, W is 0.142857142857143 - that value * 14 = 2.

Upvotes: 3

Taylor Libonati
Taylor Libonati

Reputation: 21

I am guessing that there are some unseen conditions determining if the numbers you are entering are being calculated as integers or doubles. What happens when you type out Y as Y = ((44.0 / 14.0) - (44.0 \ 14.0)) * 14.0 or however you can specify doubles in Visual Basic.

Also the bottom of this article mentions a mode that warns you when unsafe conversions take place which might help track it down. https://learn.microsoft.com/en-us/dotnet/visual-basic/programming-guide/language-features/operators-and-expressions/arithmetic-operators

Upvotes: 0

Related Questions