CODEWITHSUNDEEP
ruby-on-railsruby
martincarlin87
martincarlin87

Reputation: 11042

Ruby on Rails - BigDecimal Calculation Bug

First off, the column in the database table is a DECIMAL with a length of 10,4.

If I do this on the console or even in the controller, the result is correct:

rate = (1 / 0.7126) * 1.0883

but if I do this, it doesn't work:

rate = (1 / from_rate) * to_rate

The result is simply the value of to_rate.

from_rate and to_rate are the results of database queries and the values are correct, if I log the class using .class the result is BigDecimal, so I have no idea why this calculation doesn't work as intended.

I've tried explicitly converting using to_d and BigDecimal.new but they don't seem to make a difference.

Upvotes: 1

Views: 282

Answers (1)

SteveTurczyn
SteveTurczyn

Reputation: 36860

From evidence, it looks like the integer division instance method (invoked by / as in (1 / from_rate) returns an integer result when the argument (the divisor) is BigDecimal, so in this case it's returning the integer value 1. This isn't a problem when the argument is a float as in those cases integer division does return a float.

You may be able to circumvent this problem by just using the float divisor... do that by specifying 1.0 (a float) as the dividend.

rate = (1.0 / from_rate) * to_rate

Or alternatively you can convert the from_rate value into a float.

rate = (1 / from_rate.to_f) * to_rate

Upvotes: 1

Related Questions