CODEWITHSUNDEEP
luafloating-point
kevin lynx
kevin lynx

Reputation: 895

if lua number is double, does =/<=/>= operations always correct?

As you know, float point has a precision problem, that is a value 1 will be 0.9999999. And lua use == in C to check whether two double numbers equal. So the problem is:

a = 5.6
b = 14 * 0.4
print(a==b) ; => false

But the worse thing is:

a = 1
...
if a < 1 then print("<1") end ; => sometimes a < 1

So how can i avoid this ? I check out lua source code, it seems i can modify luai_numeq/luai_numle) macros in luaconf.h, but is this necessary ?

update

The second example is not really correct. Actually my problem is, I pass a value 1 to c/c++ which use lua_tonumber to get the value, and i store this value in a double variable, and after sometime, I push the value (lua_pushnumber) to lua, and a < 1 happened:

in lua:
my_cfunction(1)
...
in c:
int my_cfunction(lua_State *L) {
    double val = lua_tonumber(L, 1);
    ...
...
lua_pushnumber(L, val);

in lua:
local a = my_cfunction2()
if a < 1 then ... end

Upvotes: 2

Views: 2584

Answers (2)

Mud
Mud

Reputation: 28991

float point has a precision problem

It doesn't.

A 64 bit double can hold many more integer values precisely than a 32 bit integer.

Upvotes: 3

Nicol Bolas
Nicol Bolas

Reputation: 473437

As you know, float point has a precision problem, that is a value 1 will be 0.9999999

I don't know that at all. Because it's not true. 1.0 is 1.0. This:

a = 1
if a < 1 then print("<1") end

Will never print "<1". Not unless you actually change a. Even this:

a = 2
a = a - 1
if a < 1 then print("<1") end

Will likewise never hit the print statement.

As long as you are performing integer arithmetic on Lua's numbers, you will end up with integers. No addition, subtraction, or multiplication of integer values will leave you with a non-integer number value.

Upvotes: 5

Related Questions