CODEWITHSUNDEEP
jsonluafloating-pointtostringieee-754
Luatic
Luatic

Reputation: 11171

Compact yet exact floating-point number formatting

Fixing a Lua JSON implementation, I replaced "%.14g" for number formatting with "%.17g" to counter loss of precision: Formatting 17 digits using "%.17g" always suffices to allow converting an IEEE 754 double (the default Lua number type) to a string representation which can be converted back to the exact number. However, this may result in ugly number formatting: ("%.17g"):format(0.1) and ("%.17g"):format(0.3) result in "0.10000000000000001" and "0.29999999999999999" respectively due to float inaccuracies. Clearly %g doesn't always use the shortest representation of a number in case there are two decimal representations that get rounded to the same float.

I'm currently considering the following hack for cleaner (more compact) number formatting: Use %.17g only if %.16g does not produce an exact representation, e.g.:

local function number_to_string(num)
    local compact = ("%.16g"):format(num)
    if tonumber(compact) == num then
        return compact
    end
    return ("%.17g"):format(num)
end

using this, 0.1 and 0.3 are converted to their more compact representations:

> number_to_string(0.1)
0.1
> number_to_string(0.3)
0.3

tostring produces compact number representations, but does not preserve precision; besides, it seems the reference manual does not guarantee that it uses any particular format:

> tostring(0.123456789101112)
0.12345678910111
> ("%.17g"):format(0.123456789101112)
0.123456789101112

Question(s):

Upvotes: 1

Views: 135

Answers (1)

Egor Skriptunoff
Egor Skriptunoff

Reputation: 23737

will there be Lua numbers where a more compact representation exists but this fails to find it?

Yes.
Denormalized numbers might have much less than 17 significant digits:

string.format("%.17g", 2^-1074)  -- 4.9406564584124654e-324
string.format("%.16g", 2^-1074)  -- 4.940656458412465e-324
....
string.format("%.2g", 2^-1074)   -- 4.9e-324
string.format("%.1g", 2^-1074)   -- 5e-324

Single-digit-approximation gives the same number:

5e-324 == 2^-1074                -- true

Upvotes: 3

Related Questions