Reputation: 5087

How to fix precision of variable

In c#

double tmp = 3.0 * 0.05;

tmp = 0.15000000000000002

This has to do with money. The value is really $0.15, but the system wants to round it up to $0.16. 0.151 should probably be rounded up to 0.16, but not 0.15000000000000002

What are some ways I can get the correct numbers (ie 0.15, or 0.16 if the decimal is high enough).

Upvotes: 3

Answers (6)

faulty

Reputation: 8347

What you faced is a rounding problem, which I had mentioned earlier in another post

Can I use “System.Currency” in .NET?

And refer to this as well Rounding

Upvotes: 0

AaronSieb

Reputation: 8266

Use a fixed-point variable type, or a base ten floating point type like Decimal. Floating point numbers are always somewhat inaccurate, and binary floating point representations add another layer of inaccuracy when they convert to/from base two.

Upvotes: 14

spoulson

Reputation: 21591

A decimal data type would work well and is probably your choice.

However, in the past I've been able to do this in an optimized way using fixed point integers. It's ideal for high performance computations where decimal bogs down and you can't have the small precision errors of float.

Start with, say an Int32, and split in half. First half is whole number portion, second half is fractional portion. You get 16-bits of signed integer plus 16 bits of fractional precision. e.g. 1.5 as an 16:16 fixed point would be represented as 0x00018000. Or, alter the distribution of bits to suit your needs.

Fixed point numbers can generally be added/sub/mul/div like any other integer, with a little bit of work around mul/div to avoid overflows.

Upvotes: 0

Jon Skeet

Reputation: 1500405

Money should be stored as decimal, which is a floating decimal point type. The same goes for other data which really is discrete rather than continuous, and which is logically decimal in nature.

Humans have a bias to decimal for obvious reasons, so "artificial" quantities such as money tend to be more appropriate in decimal form. "Natural" quantities (mass, height) are on a more continuous scale, which means that float/double (which are floating binary point types) are often (but not always) more appropriate.

Upvotes: 5