CODEWITHSUNDEEP
vb6
user3485285
user3485285

Reputation: 1

Variables being set to wrong value

I'm having an issue with calculations being off inside my function. I'm using VB6, which may be my first problem. What's happening is I'm iterating through a loop, stepping from one variable to another variable, stepping by a variable, calculating every discount for an item. It puts each of those calculated discounts in a text file lookup table. If the value the user feeds it is between those 2 values, that's the discount it applies.

The problem comes in calculating out the discount value. Let's say I'm discounting .04 dollars per unit per step. So the first step's discount would be .04, then .08, then .12, etc. To calculate this, it counts the number of steps to go through the specified range and then multiplies the steps by the discount per step value. That's the main problem. When I do that, I get a floating point rounding error. I multiply the total steps (1) by the total discount (.04) and get .03999999999999999.

This wouldn't be a huge issue but because it calcs this out multiple times and adds calculations into calculations I can't have this loss of precision, it's throwing off my values later down the road. How do I avoid this type of rounding error? Also it's worth noting that my values that I'm assigning are doubles and singles, could this attribute to the problem?

TL;DR: 1 * .04 = .03999999999999 in VB6 for some reason. Even if I assign a floating point variable = 15.1, it becomes = 15.1000003814697. How would I go about fixing this?

Upvotes: 0

Views: 171

Answers (1)

Mark Bertenshaw
Mark Bertenshaw

Reputation: 5689

Both floating point value types (Single and Double) have this issue, due to the way they are represented in memory as two parts: a mantissa and an exponent. If you want integer-like precision, then you must use the Decimal subtype. VB6 does not support this type directly. For instance, you can't write:

Dim decValue As Decimal
decValue = 1.536

Instead, Decimal is supported via the Variant type, e.g.

Dim vdecValue As Variant
vdecValue = CDec (1.536)

The upside is you get the precision you want. The downside is that each value takes up 14 bytes rather than 4 or 8. You are also limited to functions which accept Variant/Decimals.

It is a shame that you haven't provided an algorithm. It is quite possible that you could fix the rounding problems up front. For instance, if your discount was in a known range, and a known precision, e.g., 0.11 to 0.95, you could keep track of the total discount as hundredths of a whole value. You total up the hundredths as Integer/Long values, and only convert to Single/Double at the last possible point. Obviously, this only works if you don't need a running total. Ironically, this "assumed decimal point" is exactly how VB represents Currency and Decimal values.

Upvotes: 1

Related Questions