CODEWITHSUNDEEP
powershellmath
secondplace
secondplace

Reputation: 578

PowerShell calculates wrong

By accident I came across this simple math equation which PowerShell gets wrong. 

PS 99.804-99.258
0.546000000000006

PS 10.804-10.258
0.546000000000001

This made me curious and I tried some different calculators and most Calculator gets it right and return

0.546

And:

PS 1.804-1.258
0.546

is correct too. But I found other calculators that also returned wrong results:

0.54600000000001
0.545999999999999999
0.5460000000000065

Why does this happen and how can I get the correct result in PowerShell?

Upvotes: 2

Views: 596

Answers (1)

scharette
scharette

Reputation: 9977

This is because the default type used in your example is System.Double,

99.804.GetType()

>>>>IsPublic IsSerial Name                                    BaseType
    -------- -------- ----                                    --------
    True     True     Double                                  System.ValueType

In your particular case, what you want to use is System.Decimal which is more precise,

[decimal]99.804-[decimal]99.258

>>>>0.546

or alternatively,

99.804d-99.258d

>>>>0.546

Further information

I felt the need to clarify what I meant by more precise. Basically, it is important to understand that double type has a much more larger range than decimal type. But, larger range doesn't mean more accurate.

Actually, System.Decimal was initially created to take care of currency/money problem where decimal point can be a mess. It's preciseness rely in the fact that it uses base-10 math instead of base-2 math for System.Double. Obviously, the latter is much more efficient to calculate but less accurate.

This can be validated here in the MSDN documentation where it states that,

You can use Decimal variables for money values. The advantage is the precision of the values. The Double data type is faster and requires less memory, but it is subject to rounding errors. The Decimal data type retains complete accuracy to 28 decimal places.

Floating-point (Single and Double) numbers have larger ranges than Decimal numbers but can be subject to rounding errors. Floating-point types support fewer significant digits than Decimal but can represent values of greater magnitude.

Upvotes: 5

Related Questions