CODEWITHSUNDEEP
apache-flexactionscript-3math
a.s.t.r.o
a.s.t.r.o

Reputation: 3338

Actionscript 3 Math inconsistencies

I'm trying to build a calculator in Flex / actionscript 3 but have some weird results using the class Math :

trace(1.4 - .4); //should be 1 but it is 0.9999999999999999
trace(1.5 - .5); //should be 1 and it is 1
trace(1.444 - .444); //should be 1 and it is 1
trace(1.555 - .555); //should be 1 but it is 0.9999999999999999

I know there are some issues with floating point numbers, but as you can see, it should at least fail for all of my examples, am I right?

How the problem is solved in other calculators and how should I proceed in order to build a usable calculator in Actionscript 3 please?

Thank you in advance, Adnan

Upvotes: 2

Views: 412

Answers (3)

tvanfosson
tvanfosson

Reputation: 532455

I would assume that most calculators display fewer decimal places than the precision of their floating point. Rounding to fewer decimal places than your level of precision should alleviate this sort of problem, but it won't solve all of the issues.

Upvotes: 0

Aistina
Aistina

Reputation: 12683

Your results are to be expected, and will be observed in any programming language with a floating point datatype. Computers cannot accurately store all numbers, which causes edge cases like the ones you posted.

Read up on floating point accuracy problems at Wikipedia.

Upvotes: 1

Ignacio Vazquez-Abrams
Ignacio Vazquez-Abrams

Reputation: 798626

Welcome to IEEE 754 floating point. Enjoy the inaccuracies. Use a fixed-point mechanism if you want to avoid them.

Upvotes: 4

Related Questions