CODEWITHSUNDEEP
python
Nips
Nips

Reputation: 13870

How to set precision without rounding?

I have big integers in database (numeric(24,0)). And I want to convert this number to "human readable" format. I have function:

def from_int(value, precision):
    fprec = "%." + str(precision) + "f"
    return fprec % (Decimal(value) * (Decimal(10)**(-1*precision)))

And it works:

from_int(1000000, 6)
'1.000000'
from_int(1000000, 8)
'0.01000000'
from_int(1000000, 12)
'0.000001000000'
from_int(1000000, 2)
'10000.00'

but for:

from_int(19999999999999999, 2)
'200000000000000.00'

How to set precision without rounding?

Upvotes: 2

Views: 970

Answers (3)

user234932
user234932

Reputation:

If all you want is a string representation, you may be able to get away with something like this:

>>> ('%%.%df' % 2) % 2.445
'2.44'

so in your original code:

def from_int(value, precision):
    return ('%%.%df' % precision) % value

EDIT:

To deal with precision issue, it's indeed simpler to use Decimal type, and the quantize method:

>>> d = decimal.Decimal(19999999999999999)
>>> prec = 2
>>> str(d.quantize(decimal.Decimal(10) ** -prec))
'19999999999999999.00'
>>>

ref: How can I format a decimal to always show 2 decimal places?

EDIT: Using the Decimal type, you can shift decimal places without rounding:

>>> str(d * (decimal.Decimal(10) ** -prec))
'199999999999999.99'

Upvotes: 0

Serge Ballesta
Serge Ballesta

Reputation: 148890

The problem is in the %f construct that expects a float. So even if you compute everything in decimal, you have a nasty float conversion at the end => so the rounding issue. I think you could just write :

def from_int(value, precision):
    return str((Decimal(value) * (Decimal(10)**(-1*precision))))

I tested it on your values and it gives correct results.

Upvotes: 1

Janne Karila
Janne Karila

Reputation: 25197

Formatting with %f converts the number to floating point losing precision. Instead use str:

def from_int(value, precision):
    return str(Decimal(value) * Decimal(10)**(-precision))

Upvotes: 2

Related Questions