CODEWITHSUNDEEP
type-conversiondecimalopenedgeprogress-4gldecimal-point
Nicholas Womack
Nicholas Womack

Reputation: 11

In Progress 4GL, is there a way to convert a string to a decimal without any loss in precision?

Say that I needed to turn the character variable "0.0000000001" into a decimal. But if I were to write out the following logic:

define variable tinyChar as character initial "0.0000000001" no-undo.
define variable tinyNum as decimal no-undo.

assign tinyNum = decimal(tinyChar).

display tinyNum.

It produces this result:

0.00

So that must not be the solution, and truncating would also just remove the data I'm trying to preserve. Does anyone know how I can preserve the precision of small decimal numbers? It doesn't have to be this crazy case here to the ten-billionth place, but having at least 7 or 8 numbers of precision would help with my issue.

Upvotes: 0

Views: 2019

Answers (4)

Tom Bascom
Tom Bascom

Reputation: 14020

Variables (the DEFINE VARIABLE statement) default to the maximum of 10 decimal places so you do not need to actually declare the precision.

Your problem is really that the DISPLAY format is decoupled from the internal data representation and precision (this is also true for lots of other field and variable attributes like character string width).

This is a feature. But one that people new to OpenEdge often trip over. It is very common to think that the display format constrains the storage of the field or variable. It does not. (Much to the chagrin of anyone using SQL to access the data.)

You can either change the variable's default format as Mike indicates by changing the DEFINE VARIABLE, or you can override the format in any individual DISPLAY.

So the minimum change needed to your code is:

define variable tinyChar as character initial "0.0000000001" no-undo.
define variable tinyNum as decimal no-undo.

assign tinyNum = decimal(tinyChar).

display tinyNum format "9.9999999999".

Regardless of the DISPLAY format any calculations you might do with the variable will use all 10 decimal places.

Decimal fields defined in a database schema (as opposed to variables) default to 2 decimal places. You can change that when you create them.

Calculated values assigned to a decimal field or variable will be ROUNDED to the precision defined.

Upvotes: 2

Jensd
Jensd

Reputation: 8011

Consider converting it by multiplying with a constant (1000000 or something similat). Perhaps even make it an integer? Progress will not be the best environment for such small floating point numbers. If you depend on the calculations precision will suffer!

Upvotes: -1

Mike Fechner
Mike Fechner

Reputation: 7192

Try using the DECIMALS and FORMAT options when you define the DECIMAL variable:

define variable tinyChar as character initial "0.0000000001" no-undo.
define variable tinyNum  as decimal                          no-undo
            DECIMALS 10 FORMAT "9.9999999999".

assign tinyNum = decimal(tinyChar).

display tinyNum.

Upvotes: 1

nwahmaet
nwahmaet

Reputation: 3909

Per the doc[1] the DECIMAL type stores up to 10 decimal places. If you have more, I think it truncates the value.

[1] https://docs.progress.com/bundle/openedge-abl-reference-122/page/DEFINE-VARIABLE-statement.html

Upvotes: 1

Related Questions