CODEWITHSUNDEEP
javaroundingprecisionpointfloating
tsquared
tsquared

Reputation: 119

'Practical' double rounding in Java

So I've read a lot on this, so please know that I get that a number like 0.6 can not be represented absolutely accurately as a Java double - but I do know that there is a version of a double that represents the number 0.6 closely enough that the number gets displayed as 0.6 instead of something like 0.6000000000000001 in most end interpretations of that number (e.g. including toString() or marshaling to JSON using a library like Jackson - my goal).

I'm looking for a mechanism to coerce a double into a number that is interpreted as a relative truncated precision. E.g., if I want to truncate to 0.1 precision, I want something that will represent 0.6975265613 truncated to a 0.1 precision as 0.6, not 0.6000000000000001

So, basically making the following test work:

@Test
public void testRounding() {

    // prove there is a value that displays cleanly
    Double cleanValue = 0.6d;
    assertEquals("0.6", cleanValue.toString());

    Double value = 0.6975265613d;
    Double precision = 0.1d;

    value = value / precision;
    value = value.longValue() * precision;

    // this fails with 0.6000000000000001
    assertEquals("0.6", value.toString());

}

I'm using the longValue() to truncate for speed, but you get the same result with Math.floor().

Looking at the raw double hex values, the cleanValue above is: 3fe3333333333333

While the result of the rounding (value) is: 3fe3333333333334

I need a technique that will consistently give me the intended version (the first), regardless of the rounding precision - assuming the precision doesn't push the limit of precision for the type. Conversion to String or BigDecimal will be too slow - this is an analytic app where the x10 cost of these methods will be very measurable.

Again, understood that neither hex number really represents the actual number 0.6, I just want Java to think it's 0.6 (i.e. 3fe3333333333333) :)

Thanks!

Upvotes: 3

Views: 535

Answers (1)

BevynQ
BevynQ

Reputation: 8269

This passes not sure it is what you want though

@Test
public void testRounding() {

    // prove there is a value that displays cleanly
    Double cleanValue = 0.6d;
    assertEquals("0.6", cleanValue.toString());

    Double value = 0.6975265613d;

    // get double value truncated to 1 decimal place
    value = BigDecimal.valueOf(value).setScale(1,BigDecimal.ROUND_DOWN).doubleValue();

    // this fails with 0.6000000000000001
    assertEquals("0.6", value.toString());

}

Upvotes: 1

Related Questions