CODEWITHSUNDEEP
javascriptactionscript-3floating-point
rinogo
rinogo

Reputation: 9163

Why am I seeing inexact floating-point results in ECMAScript / ActionScript 3?

Hey all, let's jump straight to a code sample to show how ECMAScript/JavaScript/AS3 can't do simple math right (AS3 uses a 'IEEE-754 double-precision floating-point number' for the Number class which is supposedly identical to that used in JavaScript)...

trace(1.1);         //'1.1': Ok, fine, looks good.
trace(1.1*100);     //'110.00000000000001': What!?
trace((1.1*100)/100);  //'1.1': Brings it back to 1.1 (since we're apparently multiplying by *approximately* 100 and then dividing by the same *approximate* '100' amount)
trace(1.1*100-110); //'1.4210854715202004e-14': Proof that according to AS3, 1.1*100!=110 (i.e. this isn't just a bug in Number.toString())
trace(1.1*100==110);   //'false': Even further proof that according to AS3, 1.1*100!=110

What gives?

Upvotes: 2

Views: 1173

Answers (4)

Matt Baker
Matt Baker

Reputation: 3754

As moonshadow states, you're running into issues with floating point precision. Floating point numbers aren't suited to the task of representing and performing arithmetic upon decimal values in the manner that you would expect. These kinds of problems are seen most often when people try to using floating point numbers for financial calculations. The wikipedia entry is good, but you might get more out of this page, which steps through an error-prone financial calculation: http://c2.com/cgi/wiki?FloatingPointCurrency

To accurately deal with decimal numbers you need a decimal library. I've outlined two BigDecimal-style libraries written in javascript that may suit your needs in another SO post, hopefully you'll find them useful:

https://stackoverflow.com/questions/744099/javascript-bigdecimal-library/1575569

Upvotes: 2

Matt Dawdy
Matt Dawdy

Reputation: 19717

Floating point inconsistencies are a known problem in many languages. This is because computers aren't designed to handle floating point numbers.

Have fun

Upvotes: 2

moonshadow
moonshadow

Reputation: 89155

Welcome to the wonderful world of floating point calculation accuracy. In general, floating point calculations will give you results that are very very nearly correct, but comparing outputs for absolute equality is unlikely to give you results you expect without the use of rounding functions.

Upvotes: 10

Paddy
Paddy

Reputation: 33867

This is just a side effect of using floating point numbers - these are binary representations of decimal numbers, there will always be some approximations.

Long explanation

Upvotes: 2

Related Questions