CODEWITHSUNDEEP
ruby
Stephen
Stephen

Reputation: 3432

Ruby BigDecimal - bug?

Something strange is going on with BigDecimal when you take the sqrt of a small number but specify a very high level of precision...

When I say strange, I mean the sqrt of 36 is 5.

Can anyone explain whats going on here or is this a bug...

require 'bigdecimal'
require 'bigdecimal/util'

@d = BigDecimal.new(36)
puts @d.sqrt(250).to_i

$ ruby1.9 test.rb 
5

$ ruby1.9 -v
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin10]

$ uname -a
Darwin jack.bidcactus.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011;

Upvotes: 0

Views: 269

Answers (3)

André Medeiros
André Medeiros

Reputation: 810

Square root of 36 is 6. When Ruby evaluates it with this precision, it gets 5.9999... As you calls to_i, you rounds down the value, getting 5 instead.

Upvotes: 3

Linuxios
Linuxios

Reputation: 35783

When you call to_i, you are removing all of the decimal precision created by the square rooting. When you call to_i, the almost infinite trail of 9s after the 5 (seen my removing to_i). If you want to do that, you might want to call round before calling to_i.

Upvotes: 0

Woot4Moo
Woot4Moo

Reputation: 24316

It returns a float. This is why you see the "strange" behavior. This is not therefore a bug. API

code:

           static VALUE
math_sqrt(VALUE obj, VALUE x)
{
    double d0, d;

    Need_Float(x);
    d0 = RFLOAT_VALUE(x);
    /* check for domain error */
    if (d0 < 0.0) domain_error("sqrt");
    if (d0 == 0.0) return DBL2NUM(0.0);
    d = sqrt(d0);
    return DBL2NUM(d);
}

Upvotes: 2

Related Questions