CODEWITHSUNDEEP
visual-foxpro
Swordblaster
Swordblaster

Reputation: 356

Visual Foxpro 9 Odd behaviour with large numeric values

Can someone please explain this behaviour and suggest a way around it?

In the command window on VFP 9.

Test 1

a = 7003602346555440

? a

Displays correct value.

Test 2

a = 7003602346555438

? a

Still fine.

Test 3

a = 7003602346555439

? a

Displays incorrect value of 7003602346555440

Test 4

? a=7003602346555439

Returns .T. as you'd expect.

Test 5

? VAL(7003602346555439)

Displays incorrect value of 7003602346555440

Clearly something odd going on with converting the numeric into the textual representation for display, but can anyone suggest a way to avoid this and ensure I always get the correct text version of the numeric?

Upvotes: 1

Views: 1128

Answers (2)

Hank Wilson
Hank Wilson

Reputation: 519

Even though the documentation says that val() will round up after 16 digits, it often rounds up at 16 and above. The example you are showing uses 16 digits and that is causing val() to round up.

Upvotes: 0

Yeshwant Mudholkar
Yeshwant Mudholkar

Reputation: 107

Source from this article

SUMMARY

Visual FoxPro is documented as having 16 digits of precision. This is an approximation: the actual maximum exactly representable number is 9007199254740992 (2^53).

MORE INFORMATION

Floating point numbers are stored in 8-byte or 64-bit representations. There are 12 bits of overhead, leaving 52 bits to store the number. There is one more implied bit that gives you 2^53 as the maximum. The maximum number that can be stored by Visual FoxPro is 2^1023. The highest power of two that is printed out exactly using the ? command with the default setting of SET DECIMALS TO 2 is 2^43.

The following code demonstrates this:

SET DECIMALS TO 2 ? 2^43 && All digits displayed
? 2^44 && Scientific notation

SET DECIMALS TO 5
? 2^53 && Maximum exact number

? 2^53 - 1 && Correct result
? 2^53 + 1 && Incorrect result: rounded in floating point

? 2^1023 && Cannot display: *'s will be printed
? 2^1022 && Can display

Upvotes: 3

Related Questions