CODEWITHSUNDEEP
luaprecision
David
David

Reputation: 21

Lua decimal precision loss

Can someone explain why in lua running:

return 256.65 * 1000000 + .000000005 - 256 * 1000000 gives 649999.99999997

whereas

return 255.65 * 1000000 + .000000005 - 255 * 1000000 and 
return 268.65 * 1000000 + .000000005 - 268 * 1000000 give 650000.0 ?

From what i can see it seems to be an issue strictly for decimal 65 (and it seems also 15) and for whole numbers within the range 256 - 267. I know this is related to doing these calculations with floating points, but I'm still curious as to what is special about these values in particular

Upvotes: 2

Views: 561

Answers (2)

Mud
Mud

Reputation: 28991

For the same reason that 10/3 is a repeating fraction in base 10. In base 3, dividing by 3 would result in whole numbers. In base 2 -- which is used to represent numbers in a computer -- the numbers you're producing similarly result in fractions that can be exactly represented.

Further reading.

Upvotes: 1

lhf
lhf

Reputation: 72312

What is special about these values is that 0.65 is not a binary fraction (even though it is a decimal fraction), and so cannot be represented exactly in floating point.

For the record, this is not specific to Lua. The same thing will happen in C.

Upvotes: 2

Related Questions