Reputation: 1177
I am doing calculation by C# on Win7.
I got a double number precision problem.
double x = 3389774.583;
string ss = "x is " + x + " x square is " + x * x + " , 11490571723552.8 - x*x is " + (11490571723552.8 - x * x) + "\n";
Console.WriteLine(ss);
you may find the result is not 0 even tough 11490571723552.8 is the x*x.
I know it is the precision problem of display and calculation.
I can set up a threshold to make the result 0.
Are there other better ways ?
Thanks
Upvotes: 0
Views: 150
Reputation: 1447
it's because square of 3389774.583 is not 11490571723552.8. If you will try windows calculator you will get value around 11490571723552.823889.
So there is no chance to get result 0 for (11490571723552.8 - x * x) where x is 3389774.583
Try
double x = 3389774.583;
double xx = x*x;
Console.WriteLine(xx);
string ss = "x is " + x + " x square is " + x * x + " , 11490571723552.8 - x*x is " + (xx - x * x) + "\n";
Console.WriteLine(ss);
Upvotes: 1