CODEWITHSUNDEEP
actionscript-3precision
Dimitris Tsatsos
Dimitris Tsatsos

Reputation: 1

as3 Number precision

I am trying to round some numbers in two decimal point and I run into a bizare behavior.

please try the following code:

var num:Number  = 30.25
for (var i = 0 ; i < 100 ; i++){
    var a:Number  = (Math.round(num * 100) / 100)
    var b:Number  =  (Math.round(num * 100) * 0.01 )
    trace (num.toString() + " -- " + a.toString() + " -- " + b.toString())
    num += 0.999;
}

x = y /100 and x = y * 0.01 should be equal.

(And x = y * 0.01 should be faster).

But if I run the above code the result is not always equal.

I get for example

while x=y/100 is always correct x=y*0.01 sometimes adds a small value like 0.000000000000004 at the end.

Am I doing something wrong? Has anyone else observed this behavior?

Upvotes: 0

Views: 995

Answers (1)

Andr&#233; Staltz
Andr&#233; Staltz

Reputation: 13994

In general, in floating point computations you should try to avoid numbers of really different magnitude in the same calculation. That's precisely the issue with these types: the point "floats", so you want to keep the point of one number of the computation close to the point of the other number.

Your question is simply put as

Why is 4623/100 == 46.23 but 4623*0.01 == 46.230000000000004?

For the specific reason, you can dig in the specific of floating point computation, for example here.

4623 is 4.623*10^3 while 0.01 is 1*10^{-3}, notice how the exponent is really different (6 orders of magnitude of difference). While 100 is just 1*10^{2}, much "closer" to 4.623*10^3.

Upvotes: 2

Related Questions