CODEWITHSUNDEEP
ruby-on-railsrails-activerecordprecision
Peter Nixey
Peter Nixey

Reputation: 16565

Prevent ActiveRecord from rounding small numerical attributes ( < 1e-20)?

I need my Rails app to work with some very small numbers i.e. less than 10e-20. To do that I need to interact between three different systems, my database (Postgres), ActiveRecord and Ruby itself. Both Ruby and Postgres seem to be happy to deal with numbers that are as small as 1e-307. However I'm struggling to get ActiveRecord to play ball.

Postgres documentation

The double precision type typically has a range of around 1E-307 to 1E+308 with a precision of at least 15 digits

Ruby documentation

MIN The smallest positive normalized number in a double-precision floating point. Usually defaults to 2.2250738585072014e-308.

So both Ruby and Postgres should be fine with numbers as small as 10e-307 and with around about 15 decimal places of precion.

Consider the following record

My 'Item' records have a rank attribute.

  # schema.rb
  create_table "items", id: :uuid, default: -> { "gen_random_uuid()" }, force: :cascade do |t|
    t.float "rank"
    ...
  end

I can see from PSQL that the value of this for one particular record is 1.24324e-20:

enter image description here

ActiveRecord is obscuring the precision

However, when I read this value via ActiveRecord it rounds it off to 1 significant figure:

myItem = Item.find('a60e5947-6e75-4c4e-8b54-c13887ad6bab')
myItem.rank
# => 0.1e-19
# (this should be 0.124324e-19)

And I can reconfirm that the precise value is in there using a raw SQL query:

ActiveRecord::Base.connection.execute("select rank from items where id = 'a60e5947-6e75-4c4e-8b54-c13887ad6bab'").values
#=> [[1.24324e-20]]

This isn't a ruby issue...

I wanted to rule out the possibility that Ruby wasn't rounding the number so I took the value that was stored in Postgres and printed it straight into the console to check I could manipulate it:

1.24324e-20 + 1.1e-20
# => 2.34324e-20 (correct)

And it's not a Rails command-line formatting issue

Since values can sometimes be obscured by the formatting Rails uses to print to the command line, I wanted to check that too. To confirm it wasn't just a formatting issue I tried multiplying the number by 10e20 and adding another number to see if the precision was just hiding somewhere in Rails' formatting:

myItem.rank 
# => 0.1e-19
i.rank * 1e20 + 1.001
# =>  2.001
# (the answer should be 2.244239)

The precision from the original number (1.34324) is ignored in the calculation. So it's not a command-line formatting issue.

Why won't ActiveRecord respect the original precision?

What do I need to do to get ActiveRecord to keep up to pace with the precision of Postgres and Ruby?

Please note: I don't want to switch the database column type

The 8-bit float column type is perfect for the numbers I wish to store. I don't need crazy precision, I just need very, very small numbers. I could switch the database column to decimal or numeric but it's a totally unnecessary amount of data to store.

Float is perfect for what I need - I just need ActiveRecord to actually read it correctly from the database....

Upvotes: 2

Views: 388

Answers (1)

Peter Nixey
Peter Nixey

Reputation: 16565

Ahem. So it turns out that during a previous database migration of the rank attribute I had assigned it a decimal type with a fixed precision.

Although I had migrated the database to using float for this attribute, I hadn't reloaded my console and so it was printing out the value as truncated by the previous decimal precision specification.

I was surprised that not only did the console need restarting but also so did my dev server, I had assumed that updated itself though apparently it didn't.

Anyway. Mystery solved. Doh.

Upvotes: 0

Related Questions