CODEWITHSUNDEEP
kdb+
user06931912
user06931912

Reputation: 39

Why is casting to float give me so many digits which are not there in the first place?

So I start with : 9.638554216867471

I have this code :

{[x;noOfDigits]
  x:string x; 

  if[null wIsDot:first ss[x;"."];:"F"$x];

  noOfChars:min(noOfDigits + 1 + wIsDot;count x);

  "F"$noOfChars#x 
  }

I expect to get this : 9.63

But I get : 9.630000000000001

"F"$"9.63" always gives 9.630000000000001

Upvotes: 0

Views: 418

Answers (3)

SJT
SJT

Reputation: 1097

Simple ways to cast a float to a string:

q)q:9.638554216867471
q)string .01 xbar q          / truncate
"9.63"
q)string .01*floor .5+100*q  / round
"9.64"

Functional alternatives to your original:

q)ft:{[x;nd]string(10 xexp neg nd)xbar x}  / truncate
q)ft[q;2]
"9.63"
q)fr:{[x;nd]string%[;s]floor .5+x*s:10 xexp nd}  / round
q)fr[q;2]
"9.64"

Upvotes: 0

terrylynch
terrylynch

Reputation: 13657

There are various built-in functions you can use if the end goal is just to pretty-print numbers to specified decimal places:

q)update prettyPrint:-27!(2i;floats) from ([]floats:9.63 1.2345678)
floats   prettyPrint
--------------------
9.63     "9.63"
1.234568 "1.23"

Other built-in functions are .Q.f and .Q.fmt but the newer -27! might be preferable

Upvotes: 1

Matthew Clarke
Matthew Clarke

Reputation: 96

In kdb the final digit in a floating point number is notoriously unreliable, you can read about it here:

https://code.kx.com/q/basics/precision/

Just to give an idea, with \P set to 0 you get:

q)"F"$"9.63"
9.6300000000000008
q)9.63="F"$"9.63"
1b

So there is nothing wrong with your code, its just a quirk to be aware of.

Upvotes: 3

Related Questions