CODEWITHSUNDEEP
luadoubleroundingsquare-root
Jakub Tětek
Jakub Tětek

Reputation: 161

Lua - round to double


The result of math.sqrt(2) seems to be irrational so this occurs:

> return  math.sqrt(2)
1.4142135623731
> return  math.sqrt(2) == 1.4142135623731
false

How do I make this "irrational" variable same as if I got the variable different way (like in the example above)?

Upvotes: 2

Views: 558

Answers (1)

Deduplicator
Deduplicator

Reputation: 45654

The variable is not irrational, it is floating-point, so it isn't even real. (the square-root of 2 is irrational though, and thus cannot be accurately represented by it)

Just use more digits for your literal, and the round-trip conversion will work. An IEEE double-precision floating-point value needs 17 significant decimal digits to safely represent it, not 14.

Let's see what happens when we take the number 1 and uptick it in the least significant bit. (The '0x' means the numeral is hexadecimal. That makes it easier for me to control the bits for this example.):

x = 0x1.0000000000001
> print(x == 1)
false
> print(('%.16g'):format(x))
1
> print(('%.17g'):format(x))
1.0000000000000002

Upvotes: 4

Related Questions