Why do calculations with decimal numbers sometimes yield inaccurate results.
search cancel

Why do calculations with decimal numbers sometimes yield inaccurate results.


Article ID: 71337


Updated On:


Gen Gen - Workstation Toolset Gen - Host Encyclopedia Gen - Run Time Distributed


Some Gen calculations with decimal numbers can produce inaccurate results. What can cause this?


Release: KGNDDL99000-8.5-Gen-DBP Developer License


Any time a floating-point data type (float or double) is used, the possibility of a loss of precision exists because of the way floating-point numbers are stored. A loss of precision is possible even if there is no mathematical operation on the floating-point data types, i.e. just storing a number in a floating-point data type can be enough for a loss of precision to occur.

Gen generates floating-point data types in either of the following cases:

  1. An attribute is defined with decimals
  2. An attribute has length greater than 9 digits.

Therefore to ensure numeric accuracy, it is recommended to turn on the Decimal Precision property for any numeric attribute that meets either of the following two conditions:

  1. Number of decimal places > 0
  2. Length of attribute is > 9 (even if number of decimal places = 0)

If an attribute is declared with a length of 8 and with 2 decimal places (8,2), that meets condition 1 above, so Decimal Precision should be checked.
If an attribute is declared as a 10 digit integer (10,0), that meets condition 2 above, so Decimal Precision should be checked.

In addition calculations should be set to ROUNDED to ensure no loss of accuracy.