Reputation: 4817
This query
range x from -1 to 1 step 0.2
| project x
produces this result. Can this be correct?
Upvotes: 2
Views: 481
Reputation: 5298
Working with numbers with a floating point can’t be expected to be 100% precise.
I wrote a quick C program that starts with float f = 0.0f
and then runs f += 0.1
100 times, while printing f every 10 iterations, and the result is:
1.000000
2.000000
2.999999
3.999998
4.999998
5.999997
6.999996
7.999995
8.999998
10.000002
You can see/modify/run see the code I wrote here: https://onlinegdb.com/HyylownHd
The lack of precision may happen not only in the arithmetic calculations, but also in the code that converts the float number to a string (to pass between layers or just before printing it).
As @avnera suggested, using decimal
indeed gives you better precision, but you still shouldn't expect it to be 100% precise.
Upvotes: 2
Reputation: 7608
Yes, the type that is inferred based on the values provided is double (real) which is a floating point. If you would like to see precise numbers use the decimal type, for example:
range x from decimal(-1) to decimal(1) step decimal(0.2)
Upvotes: 2