CODEWITHSUNDEEP
javascriptfirefoxroundingprecision
Dave Morgan
Dave Morgan

Reputation: 61

Firefox and Javascript Rounding Rules

I don't know if I'm missing something obvious here but...

In IE, Opera and Chrome, I get what I expect from rounding numbers ending in a 5:

125 toPrecision(2) => 130
11.5 toPrecision(2) => 12

This is what I'd expect.

Firefox, though, is a little more 'sophisticated' yielding the following:

125 toPrecision(2) => 120 //wtf!!!
11.5 toPrecision(2) => 12

After a bit of head scratching, I've come to the conclusion that Firefox is using a 'rounding even' rule where, if the digit before the 5 is even the number rounds down and if the digit before the 5 is odd the number rounds up:

0.5 => 0
1.5 => 2
2.5 => 2
3.5 => 4, etc.

I am using the rounded results to test student solutions to engineering questions with pseudo-randomly generated question inputs. A question input in Chrome could be h=1020 mm but h=1030 mm in FF, Chrome or Opera.

I need a function to make rounding consistent, i.e. I want 0.0001235 to round up to 0.000124 and I want 1234 to round to 1240 so I can't use a simple num = Math.floor(num + 0.5); To complicate matters a bit, I want input variables and student answers to be correct to 3 sig digs unless the first digit is a 1, in which case I want 4 sig digs:

234.5 => 235
134.5 => 134.5

I've hacked a solution to the 3 or 4 sig digs depending upon the first digit by converting the number to a string and testing the first non-zero, non-decimal point and non-negative character for '1' - not pretty, but it works. I could do something similar for the rounding problem, checking whether the digit to be rounded is a 5 but I'm wondering if there is an elegant bit-wise solution.

Upvotes: 6

Views: 997

Answers (1)

vinc17
vinc17

Reputation: 3476

About "I want 0.0001235 to round up to 0.000124": This does not make much sense since unlike 0.5, 1.5, 2.5, etc., the decimal number 0.0001235 isn't exactly representable in radix 2; so its rounding may be a little above (A) or a little below (B) the exact value, where (A) and (B) would yield different roundings: 0.000124 for (A), 0.000123 for (B).

Even if you get a fractional number such as 1.5 to be rounded to an integer after some computations, (1.5).toPrecision(2) will not necessarily yield the value that is the closest to the exact value. The reason is that the exact value may be just below 1.5, in which case the correct rounding is 1, not 2. This problem is known as the Table Maker's Dilemma.

This also means that if two students use slightly different methods (both rather accurate) and round their approximations, they will get different rounded results (even though their approximations could be very close to each other). In case of a tie such as 0.0001235 (to be rounded to 0.000124), the Table Maker's Dilemma is unavoidable unless all computations are carried out exactly or the algorithm specifically checks tie cases.

However if the exact result is a tie or very close to a tie, it is still possible to check whether the student has answered either (A) or (B).

Upvotes: 0

Related Questions