|
View previous topic :: View next topic
|
| Author |
Message |
bhairon singh rathore
New User
Joined: 19 Jun 2008
Posts: 91
Location: banglore
|
|
|
|
Hi all,
My table has a col defined as float and having value 0.1.
After fetching from the table i am moving this value to a comp-2 variable in my program. the comp-2 variable is part of an array.
But here 0.1 is getting converted to 0.10000000000000001 ... the 17th position is having a 1, due to which computations are coming wrong.
Can anyone help ?
Thanks,
Bhairon |
|
| Back to top |
|
 |
CICS Guy
Senior Member
Joined: 18 Jul 2007
Posts: 2146
Location: At my coffee table
|
|
|
|
| Why comp-2? floating point will almost always have a small difference in the least significant digits. Why not half round and move to a comp-3 field? |
|
| Back to top |
|
|
Robert Sample
Global Moderator
Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA
|
|
|
|
Ah, the days of numerical analysis in college ... long ago yet still relevant. The decimal value 1/10 or .1 cannot be exactly represented as a floating point binary number. The computer generates the best approximation it can but the number's not going to be exact no matter what. Any time floating point is used, representation errors occur and need to be dealt with.
You can convert the value to a fixed number as CICS guy suggested (if possible -- not all floating point values can be so represented) and work with the fixed value.
If you cannot convert the value, then either rounding or truncation will be required in your coding to adjust values for the inherent imprecision of floating point.
The classic test is to use this code:
| Code: |
05 I PIC 9(02) VALUE 1.
05 INCR-LOW-TO-HIGH USAGE COMP-2 .
05 INCR-HIGH-TO-LOW USAGE COMP-2 .
05 MULT-LOW-TO-HIGH USAGE COMP-2 .
05 MULT-HIGH-TO-LOW USAGE COMP-2 .
LINKAGE SECTION.
/
PROCEDURE DIVISION.
S1000-MAIN SECTION.
MOVE 1 TO INCR-LOW-TO-HIGH
MULT-LOW-TO-HIGH
MULT-HIGH-TO-LOW.
MOVE 30 TO INCR-HIGH-TO-LOW.
PERFORM
VARYING I FROM 1 BY 1
UNTIL I > 30
COMPUTE MULT-LOW-TO-HIGH = MULT-LOW-TO-HIGH *
INCR-LOW-TO-HIGH
ADD 1 TO INCR-LOW-TO-HIGH
COMPUTE MULT-HIGH-TO-LOW = MULT-HIGH-TO-LOW *
INCR-HIGH-TO-LOW
SUBTRACT 1 FROM INCR-HIGH-TO-LOW
END-PERFORM.
DISPLAY 'MULT LOW TO HIGH ' MULT-LOW-TO-HIGH.
DISPLAY 'MULT HIGH TO LOW ' MULT-HIGH-TO-LOW. |
which produces this output:
| Code: |
MULT LOW TO HIGH .26525285981219099E 33
MULT HIGH TO LOW .26525285981219102E 33 |
which shows that 1 times 2 times ... 29 times 30 is not the same as 30 times 29 times ... 2 times 1 on a computer. |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
What is the content of the field? If it is a quantity or an amount, it should probably not be defined as float. . . |
|
| Back to top |
|
|
|
|
