Reputation: 11
How is R deciding to convert 0.099999999999999 into 0.1?
> format(0.3/3, digits=17)
[1] "0.099999999999999992"
> format(0.3/3, digits=16)
[1] "0.09999999999999999"
> format(0.3/3, digits=15)
[1] "0.1"
Upvotes: 1
Views: 163
Reputation: 26185
The full decimal expansion of 0.03/3 in IEEE 754 64-bit arithmetic is:
0.09999999999999999167332731531132594682276248931884765625
If the most significant dropped digit is the first 6 it rounds up, giving:
0.099999999999999992
If the most significant dropped digit is the first 1, it rounds down, giving:
0.09999999999999999
If the most significant dropped digit is the last 9 of the block of nines, it rounds up, giving:
0.1
All your results, including the final 0.1, can be explained by assuming decimal rounding to the number of digits you specified.
Upvotes: 1