CODEWITHSUNDEEP
javac#.netvb.netcompilation
Polynomial Proton
Polynomial Proton

Reputation: 5135

Visual Studio Wrong output for float/single operation

I'm trying to understand how floating point operations give different output. The tests performed below result to 1; however vb.net and c# give different output, whereas java gives a different output. It probably has to do something with compiler and I read What Every Computer Scientist Should Know About Floating-Point Arithmetic but its confusing, can someone explain in simple language?

VB.NET

Dim x As Single = 1.000001
Dim y As Single = 0.000001

Dim result = x - y

Output: 0.9999999 Click here to see output Same goes for C#

Also, while watching the variable in visual studio, the value of result is different from what it outputs, which is trimmed while printing and only seven 9's are printed(which is understood), but I dont understand how the actual result for them is 0.99999994 VS floating operation output

Update: Alright, I'm more specifically interested in how this calculation is done(removed java stuff)

Upvotes: 0

Views: 303

Answers (1)

Nader
Nader

Reputation: 152

Numbers in visual basic Single (or c# float) are stored as IEEE 754-2008. The number is stored in 32 bits. first bit is the sign the next 8 bits store the exponent and the next 23 bits store the fraction.

First the integer part of the number is converted to base 2. then the fraction is converted to base 2 and the number is shifted with the right exponent to matches this format:

1.x1..x23 x 2^e

where x1 to x23 are the bits in the fraction part and e is the exponent.

For example 0.25 is converted to: 1.0 x 2^-2

Note that the significant digits are limited to 23 bits.

In your example 1.000001 is converted to 1.<20 zeors>101001... however we can only take the first 23 digits (1.<20 zeors>101). However for the 0.000001 we can start the 23 digits from the first 1 (which is bit 20th) and use the exponent -20 and store the number in much higher precision.

Upvotes: 2

Related Questions