CODEWITHSUNDEEP
c#mathprecision
user2154014
user2154014

Reputation: 3

C# decimal precision function

I have a decimal function:

public decimal NGD(long TotalPages, long XHits, long YHits, long BothHits)
{
    return ((Math.Max(XHits, YHits)-BothHits)/(TotalPages - Math.Min(XHits, YHits)));
}

upon printing the arguments I get NGD(44000000000,36100000,5630000,3580000) which translates to return ((36100000 - 3580000 )/(44000000000 - 5630000)); yet in my application I get a 0 when it should be 0.00073918549

Upvotes: 0

Views: 490

Answers (3)

Jeppe Stig Nielsen
Jeppe Stig Nielsen

Reputation: 61912

All of your arithmetic is done with long (System.Int64) which is an integer type. Only after the division is finished (giving an integer result) will the quotient be converted to decimal.

Consider casting your dividend (or even just the leading term of it) to decimal (System.Decimal). Then all other long values will automatically be "promoted" to decimal along the way.

Of course you may instead change one parameter (or all parameters) to have type decimal. No explicit cast will be needed to convert an expression of type long into an method argument of type decimal.

Upvotes: 0

JMCF125
JMCF125

Reputation: 388

That's because you're using integer division, which truncates the result. Use return ((decimal)(36100000 - 3580000 )/((decimal)(44000000000 - 5630000))); for decimal division.

Upvotes: 2

SLaks
SLaks

Reputation: 887215

You're doing integer division, which can only return a truncated integer.

You need to cast one operand to a non-integral type (probably decimal).

Upvotes: 2

Related Questions