CODEWITHSUNDEEP
mysqlsql
Alexander Schranz
Alexander Schranz

Reputation: 2450

SQL inaccurate numbers

I have a sum of bit masks which is (2^63)-1. But the result of this is inaccurate. The last 3 numbers are always 0 or replaced with the e18 style.

Simple example SQL:

SELECT
    POW(2,63), /* 9.223372036854776e18 */
    CAST(POW(2,63) AS DECIMAL(65)), /* 9223372036854776000 */
    (POW(2,63) - 1), /* 9.223372036854776e18 */
    CAST((POW(2,63) - 1) AS DECIMAL(65)) /* 9223372036854776000 */

Upvotes: 2

Views: 130

Answers (1)

cliffordheath
cliffordheath

Reputation: 2606

POW() returns a "double precision" floating point number, which only has a 53-bit mantissa. You're expecting accuracy to 64 bits. See http://dev.mysql.com/doc/refman/5.7/en/floating-point-types.html

Upvotes: 5

Related Questions