CODEWITHSUNDEEP
vb.netvisual-studio-2012
Zack
Zack

Reputation: 117

Visual Basic .Net Rounding Issue

I am building a unit converter with Visual Basic and Visual Studio 2012. I am using the conversion rate of 1 inch = 0.0833333 feet for the inch to foot conversion. When I type in 12 inches, it gives me an answer of 0.9999996 instead of 1. How can I fix this problem?

The issue is with the inch to foot conversion below.

            ' converts inch to...
            If cbo1.SelectedIndex = 3 Then
                If cbo2.SelectedIndex = 0 Then
                    ' meter
                    txtUnit2.Text = (dblUnit1 * 0.0254).ToString.Trim
                ElseIf cbo2.SelectedIndex = 1 Then
                    ' millimeter
                    txtUnit2.Text = (dblUnit1 * 25.4).ToString.Trim
                ElseIf cbo2.SelectedIndex = 2 Then
                    ' foot
                    txtUnit2.Text = (dblUnit1 * 0.0833333).ToString.Trim
                ElseIf cbo2.SelectedIndex = 3 Then
                    ' inch
                    txtUnit2.Text = txtUnit1.Text
                End If
            End If

Upvotes: 1

Views: 205

Answers (4)

Zack
Zack

Reputation: 117

Here is my final, working code.

                            ' converts inch to...
            If cbo1.SelectedIndex = 3 Then
                If cbo2.SelectedIndex = 0 Then
                    ' meter
                    txtUnit2.Text = (dblUnit1 * 0.0254).ToString.Trim
                ElseIf cbo2.SelectedIndex = 1 Then
                    ' millimeter
                    txtUnit2.Text = (dblUnit1 * 25.4).ToString.Trim
                ElseIf cbo2.SelectedIndex = 2 Then
                    ' foot
                    txtUnit2.Text = (dblUnit1 * (1 / 12)).ToString.Trim
                ElseIf cbo2.SelectedIndex = 3 Then
                    ' inch
                    txtUnit2.Text = txtUnit1.Text
                End If
            End If

Upvotes: 0

vbigham
vbigham

Reputation: 191

You could try calculating your conversion number at runtime, you shouldn't lose any precision that way..

    Dim twelveInches As Double = 12.0
    Dim oneInchValue As Double = 1.0 / twelveInches
    Console.WriteLine(oneInchValue.ToString())
    Console.WriteLine(String.Format("{0} x {1} = {2}", oneInchValue, twelveInches, oneInchValue * twelveInches))

Output:

0.0833333333333333
0.0833333333333333 x 12 = 1

Upvotes: 3

Hanlet Escaño
Hanlet Escaño

Reputation: 17380

If you want a rounded result, use Math.Round:

Console.WriteLine(Math.Round(12 * 0.0833333))

prints 1.

Upvotes: 1

Joel Coehoorn
Joel Coehoorn

Reputation: 415881

When I type in 12 inches, it gives me an answer of 0.9999996 instead of 1

0.0833333 * 12 = 0.9999996

Put that into Windows Calculator and you'll get the same result. What did you expect?

Try defining your constant with an expression:

Const inch As Decimal = 1D / 12D

Upvotes: 2

Related Questions