CODEWITHSUNDEEP
c#linq-to-sqlfloating-point
Gregoire
Gregoire

Reputation: 24832

linq to sql strange float behaviour

I have a problem with a double (TaxRate here), that do not have the same value when it pass through functions:

I first call a function

DataContext.UpdateCategory(theCategorie.Code, theCategorie.Description, theCategorie.TaxRate, theCategorie.TaxCode);

When I run the debugger the value of TaxRate is 19.6

Then when I put a stop point in the dbml function that is called:

 [Function(Name="dbo.UpdateCategory")]
 public int UpdateCategory([Parameter(Name="Code", DbType="VarChar(20)")] string code, [Parameter(Name="Description", DbType="NVarChar(512)")] string description, [Parameter(Name="TaxRate", DbType="Float")] System.Nullable<double> taxRate, [Parameter(Name="TaxCode", DbType="NChar(10)")] string taxCode)
 {
     IExecuteResult result = this.ExecuteMethodCall(this, ((MethodInfo)(MethodInfo.GetCurrentMethod())), uNSPSC, description, taxRate, taxCode);
     return ((int)(result.ReturnValue));
 }

here the value is 19.6000003814697. You see the strange added decimal? there are no operations between these two calls, so why these decimals appears?

Upvotes: 1

Views: 2845

Answers (1)

KristoferA
KristoferA

Reputation: 12397

Is theCategorie.TaxRate a float? If so, you're assigning a float into a double. The extra decimals is due to higher precision in the double, so the double's nearest to a float's 19.6...

The output of this illustrates it:

float f = 19.6F;
double d = f;
double d2 = 19.6D;

System.Diagnostics.Debug.WriteLine("Float: " + f.ToString());
System.Diagnostics.Debug.WriteLine("Double from float: " + d.ToString());
System.Diagnostics.Debug.WriteLine("Double: " + d2.ToString());

As a workaround, you can either change the type of theCategorie.TaxRate to a double, or if you want to keep it as a float you can cast & round it when calling the stored proc:

DataContext.UpdateCategory(theCategorie.Code, theCategorie.Description, Math.Round((double)theCategorie.TaxRate, 6), theCategorie.TaxCode);

Upvotes: 1

Related Questions