CODEWITHSUNDEEP
sql-serverdecimaldivision
Abdalwhab Bakheet
Abdalwhab Bakheet

Reputation: 1163

SQL Server Decimal Operation is Not Accurate

When I run this simple operation in SQL server:

Select 800.0 /30.0

I get the value 26.666666, where even if it rounds for 6 digits it should be 26.666667. How can I get the calculation to be accurate? I tried to search about it online and I found a solution where I cast each operand to a high precision decimal before the operation, but this will not be convenient for me because I have many long complex calculations. think there must be a better solution.

Upvotes: 0

Views: 1017

Answers (3)

seanb
seanb

Reputation: 6685

I believe it's because SQL Server takes your numbers as decimal values (which are exact e.g., 6.6666 and 6.6667 means exactly those values, not 6 and two-thirds) rather than float values (which can work with approximate numbers).

If you explicity cast/convert it to a float at the start, you should get your calculations running smoothly.

Here's some examples to demonstrate the difference between int, decimal, and float calculations

  • Dividing 20 by 3
  • Dividing 20 by 3, then multiplying by 3 again (which mathematically should be 20).
SELECT  (20/3)                             AS int_calc,
        (20/3) * 3                         AS int_calc_x3,
        (CAST(20 AS decimal(10,3)) /3)     AS dec_calc,
        (CAST(20 AS decimal(10,3)) /3) * 3 AS dec_calc_x3,
        (CAST(20 AS float) /3)             AS float_calc,
        (CAST(20 AS float) /3) * 3         AS float_calc_x3

with the following results

int_calc   int_calc_x3   dec_calc   dec_calc_x3   float_calc         float_calc_x3
6         18             6.666666   19.999998     6.66666666666667   20

In your case, you can use

Select CAST(800.0 AS float) /30.0

which results in 26.6666666666667

Note if you then multiply back by 30, it gets the correct result e.g.,

Select (CAST(800.0 AS float) /30.0) * 30

results in 800. Solutions dealing with decimals will not have this.

Note also that once you have it as a float, then it should stay a float until converted back to a decimal or an int somehow (e.g., saved in a table as an int). So...

SELECT A.Num / 30
FROM (Select ((CAST(800.0 AS float) /30.0) * 30) AS Num) AS A

will still result in 26.6666666666667

This will hopefully help you in your long complex calculations.

Upvotes: 0

CR241
CR241

Reputation: 2613

This might helpful:

Use ROUND (Transact-SQL)

SELECT ROUND(800.0 /30.0, 5) AS RoundValue;

Result:

RoundValue

26.666670

Upvotes: 0

Thom A
Thom A

Reputation: 95588

When a using division, in SQL Server, any digits after the resulting scale are truncated, not rounded. For your expression you have a decimal(4,1) and a decimal(3,1), which results in a decimal(10,6):

Precision = p1 - s1 + s2 + max(6, s1 + p2 + 1)
Scale = max(6, s1 + p2 + 1)

As a result, 26.66666666666666~ is truncated to 26.666666.

You can get around this by can increasing the size of the precision and scale, and then CONVERT back to your required precision and scale. For example, increase the precision and scale of the decimal(3,1) to decimal(5,2) and convert back to a decimal(10,6):

SELECT CONVERT(decimal(10,6),800.0 / CONVERT(decimal(5,3),30.0));

This returns 26.666667.

Upvotes: 2

Related Questions