CODEWITHSUNDEEP
ruby
Bruce
Bruce

Reputation: 407

Accuracy of nanosecond component in Ruby Time

I have a duration in seconds with millisecond precision, 420.854 (420 seconds and 854 milliseconds).

I want to format it like (Minutes):(Seconds).(Milliseconds) so I tried this:

Time.at(421.854).utc.strftime('%M:%S.%L')

The result: "07:01.853" As you can see it's off by one millisecond.

So I dug into it further:

Time.at(421.854).nsec # Output: 853999999 (expected 854000000)

I believe the unexpected output of the above is the cause of the problem. Some numbers work as expected though for example:

Time.at(421.855).nsec # Output 855000000 as expected

Does anyone have any suggestions on solving this, the best I can come up with is to "fix" the nanosecond component afterwards with (nanoseconds.to_f / 1000000).round * 1000000 but I feel like there might be a better approach. Really appreciate any thoughts on the subject.

Thanks.

Upvotes: 3

Views: 1603

Answers (1)

Jon Skeet
Jon Skeet

Reputation: 1503090

I strongly suspect this is due to Time.at taking a binary floating point number - that it has nothing to do with time at all.

I'm not a Ruby dev, but in IEEE-754 64-bit floating point types, the closest value to 421.854 is exactly

421.85399999999998499333742074668407440185546875

... so my guess is that that exact value is being truncated, effectively.

(The nearest 32-bit floating point value is 421.85400390625.)

If you use a Rational instead to specify it as effectively an integer number of milliseconds (Rational(421854, 1000)), you may well find that works. Alternatively, use at(seconds, microseconds_with_frac):

Time.at(421, 854000).utc.strftime('%M:%S.%L')

Upvotes: 6

Related Questions