CODEWITHSUNDEEP
vba
Bob Jansen
Bob Jansen

Reputation: 1287

Error taking int of logs in VBA

When I calculate log(8) / log(2) I get 3 as one would expect:

?log(8)/log(2)
3

However, if I take the int of this calculation like this the result is 2 and thus wrong:

?int(log(8)/log(2)) 
 2

How and why does this happen?

Upvotes: 0

Views: 78

Answers (1)

Linger
Linger

Reputation: 15058

Likely because the actual number returned is of type double. Because floats and doubles cannot accurately represent most base 10 rational numbers the number returned is something like 2.99999999999. Then when you apply int() the .999999999 is truncated.

How floating-point number works: it dedicates a bit for the sign, a few bits to store an exponent, and the rest for the actual fraction. This leads to numbers being represented in a form similar to 1.45 * 10^4; except that instead of the base being 10, it's two.

Upvotes: 3

Related Questions