CODEWITHSUNDEEP
excelfloating-pointdoublerounding-errorvba
James Baker
James Baker

Reputation: 333

Excel VBA rounding with doubles

I am trying to do a very accurate calculation using doubles that relate to cells in my worksheet. As I test I wrote the following code:

Sub MacroTest()

Dim opt As Worksheet
Set opt = Sheets("Optimisation")

Dim test As Double

test = CDec(opt.Cells(2, 10))

End Sub

In the cell J2 is the decimal value -£3,298.8599993... however when debugging the code it is clear that the test double is only picking up -3298.86 as the value. I don't understand why this would be as I thought doubles were more accurate than this. Most of the posts I have read seem to think that using CDec should sort this problem out, however as shown this does not seem to work for me.

Note - The value in J2 has been calculated using a formula, not sure if this makes any difference. Any help is greatly appreciated - thanks.

Upvotes: 1

Views: 1359

Answers (2)

CoRrRan
CoRrRan

Reputation: 111

I can't comment yet, but here is what is wrong with your code:

You are using a CDec-conversion formula on a double-datatype, and the result from this function is put into a double-datatype again. So you do not gain anything here accuracy wise.

Are you sure you want to use the double-datatype, while you could use the decimal-datatype in VBA?

In my example, cell "A1" contains the decimal number 3298.859999.

Dim varDec As Variant
varDec = CDec(Range("A1").Value2)

Debug.Print "Value in A1 : " & Format(varDec, "0.0000000000000000000000")

Direct casting of a variable to the Decimal-datatype is not possible in VBA, but the Variant-datatype has the Decimal-datatype as a subtype.

To see this for yourself: look in the "Locals"-window in the VBE while debugging. It will show:

Expression : var
Value      : 3298.859999
Type       : Variant/Decimal

Background article on Excel numeric precision

Upvotes: 0

Axel Richter
Axel Richter

Reputation: 61852

You should use opt.Cells(2, 10).Value2 in this case. Range.Value2 property:

The only difference between this property and the Value property is that the Value2 property doesn’t use the Currency and Date data types. You can return values formatted with these data types as floating-point numbers by using the Double data type.

So the default .Value will return Currency if the cell is formatted as currency while .Value2 will return a Double nevertheless.

Upvotes: 3

Related Questions