CODEWITHSUNDEEP
c#division
MSGrimpeur
MSGrimpeur

Reputation: 1

c# division giving odd result on multiple calls

We have an class responsible for converting internal units to display units. So if our display units are m and internal units are mm it would divide the internal units by a conversionFactor of 1000. The user can add entities into the system at a varying x,y,z co-ordinates. We have an odd occurrence where a user is inputting units at 1000 mm so the display is showing 1m. the input is consistently 1000 mm but every now and again the division of 1000/1000 seems to be throwing up .9999999m instead of 1m. so in our grid we have 1m,1m,1m,1m,0.9999m,1m,1m etc. Sometimes the .9999m never appears some times it is straight away sometimes it is occurs after 20 to 100 inputs. We are investigating if something odd is happening on the input side, but I wondered if anyone else has come across something like this?

I should say we are converting it to a string to display.

Upvotes: 0

Views: 264

Answers (3)

MsGrimpeur
MsGrimpeur

Reputation: 1

Thanks for all your help we have tracked it down to a weird side effect from inputing a different object.

The issue is that if a different object is inserted by any multiple of 3 times the error is triggered e.g. objectA is input 3 times at 1m all okay, then after this objectB is input at 1m 0.9999m appears however if objectA is input 1,2,4 or 5 times there is no problem. 6 times and the problem reappears, 9 times etc. What fun we have.

Upvotes: 0

Adam P
Adam P

Reputation: 4703

If the two numbers you're dividing are floating-point values (i.e. double, float, decimal) than you may be experiencing a rounding error. Try changing them to non-floating types if possible and try to see if you can replicate the problem.

Upvotes: 1

fbl
fbl

Reputation: 2910

I'm guessing it's a display thing... what happens when you format the string to say... 9 decimal places?

var str = string.format("{0.000000000}", funkyVal);

I'd ask this via comment, but apparently I'm not a high enough level ;(

Upvotes: 0

Related Questions