CODEWITHSUNDEEP
objective-c
etayluz
etayluz

Reputation: 16416

Converting int to double screws up the decimal point

In the debug window, when I input this command:

po 1912/10.0

The output is 191.19999999999999.

What I really want to get back is 191.2.

Why is this happening, and how can I convert an int into a double with precision?

Upvotes: 0

Views: 1378

Answers (3)

Aaron Brager
Aaron Brager

Reputation: 66242

From What Every Programmer Should Know About Floating-Point Arithmetic:

Why don’t my numbers, like 0.1 + 0.2 add up to a nice round 0.3, and instead I get a weird result like 0.30000000000000004?

Because internally, computers use a format (binary floating-point) that cannot accurately represent a number like 0.1, 0.2 or 0.3 at all.

When the code is compiled or interpreted, your “0.1” is already rounded to the nearest number in that format, which results in a small rounding error even before the calculation happens.

This is why programmers say you should only ever store money as an integer. For example int cents = 1995; rather than float dollars = 19.95.

If your app doesn't need to be 100% precise (for example, if you're calculating screen coordinates or translucency or a color) just format your float rounded to 1 or 2 decimal places:

double someValue = 1912/10.0;
NSLog(@"2 decimals: %.2f", someValue);
NSLog(@"0 decimals: %.0f", someValue);

This code will output:

2 decimals: 191.20
0 decimals: 191

Upvotes: 3

AixNPanes
AixNPanes

Reputation: 1260

Let me try to explain this another way. When you express a number with a fractional part with a float or double, precision is most often lost. There's no way around that. If you store 1912 as a float and store 10 as a float then divide the first stored value by the second, the value will NEVER be 191.2. That's just the way floating point numbers work. If you look at the number in a debugger you'll see something like 191.19999999999999 as you describe. This, in itself, is an approximation as the value should be 191.19999999999999... but of course you can't even type all the digits in the decimal value of that stored result as the number of digits approaches infinity.

If you're going to use floating point, that's what you'll get. No way around it.

If you really want to get 191.2, then you can't use floating point, at least without doing rounding. Instead, you need to normalize the numbers by just storing the value as 1912 and printing the value with a decimal point to the left of the 2.

There's another brief online description at http://floating-point-gui.de/basic/

Upvotes: 0

AixNPanes
AixNPanes

Reputation: 1260

That's normal for a floating point number. Double is obviously just an extended precision floating point number. If you want to keep the pristine decimal digits, then don't allow any float/double conversion. Instead store the result as a scaled integer (in your case 1912) and place the decimal manually.

Upvotes: 0

Related Questions