CODEWITHSUNDEEP
c#decimal
Arsen Zahray
Arsen Zahray

Reputation: 25287

How to deal with decimal operation error?

I have a scaling function for values, which looks something like this:

r = a/b*c;

Where r,a,b,c are decimals.

Most of the times, it works correctly, but sometimes I get something like this:

1.5M/3.09785668M*3.09785668M => 1.5000000000000000000000000001M

This is a problem for me because over time in my calculations, the error escalates.

What is the best way to detect and remove this kind of errors?

Upvotes: 0

Views: 236

Answers (1)

grek40
grek40

Reputation: 13438

Detect

r = a/b*c;
Debug.Assert(r*b/c == a, "Not good");

Avoid non-terminating sub-result but still be vulnerable to precision loss and non-terminating end results

r = a*c/b;

So basically, the two kinds of problematic sub-results are (1) finite precision loss and (2) non-terminating numbers

(1):

var loss1 = 1.000000000000000000001m * 1.000000000000000000001m;
// should be 1.000000000000000000002000000000000000000001m
// will be be 1.000000000000000000002m

(2):

var loss2 = 1m/3m;
// should be 0.333333333333333333333.. infinite line of 3
// will be be 0.3333333333333333333333333333M

The first kind can happen on multiplication and division, the second kind only on division. The first kind can be fixed by adding precision, the second kind can only ever be fixed by carrying the fraction as is instead of trying to create a single real number of any limited precision.

Upvotes: 1

Related Questions