We made a change to one of our sort step on May and missed the EQUALS parameter in sort step. Last week, one of the successor job has failed due to duplicate data. I do not have any i/p & o/p files available to say what data has created/used. I found one of the successor job input file has data in a random order under its key. So, when I checked this sort card I noted that we missed EQUALS parameter.
I verified last few runs of our job and in SYSOUT ICEXXXI message options EQUALS=N. Could someone tell me how sort has worked without EQUALS option for this time? IS there any other parameter controls that? Now I modified the sort to include EQUALS, but I would like to understand why we had issues all of a sudden.
Joined: 28 Jan 2012 Posts: 316 Location: Room: TREE(3). Hilbert's Hotel
Probably because your site has EQUALS=NO as the default.
And specifying EQUALS in your SORT card temporarily override the default option, so that equally collating records are written in their original order for your job.
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
With EQUALS the order of duplicate keys on the output from a SORT or MERGE is predictable. For SORT they are in the same order as the input file, for MERGE they are in the same order as on each successive SORTINGnn file.
With NOEQUALS, the order is not predictable, but is whatever happens to be the most convenient when the next record is written.
Both EQUALS and NOEQUALS work. Day in, day out, every time.
Notice that "not predictable" is not the same as "random".
Quote:
Last week, one of the successor job has failed due to duplicate data
If this is an accurate statement, you can't even think to blame SORT for that.
In fact, you can't blame SORT anyway. If it was coded wrong and no-one got it in what should have been several stages of testing, long for blame elsewhere.
Until you explain otherwise, your search for why you had a failure has discovered and unconnected error elsewhere.
You'd best describe the failure and the relevant data in more detail.
We tested it so many times but we didn't notice any change. This code has been in production and running from May and issue was noticed only on August last week. Our business questions why it took so long to create an issue? I guess that's a valid question.
In my previous shop, if I don't use EQUALS, i will see a difference immediately, but here I do not see that. I cross checked whether there is any default system OPTION as EQUALS=Y, but ICEXXXI message states clearly that EQUALS=N in all previous run.
The data which we had is in above format where column 7-8 is record type, 11 - header 32 - detail and 23 - trailer. It needs to be in that order. So, EQUALS is a must when we sort on first two fields.
However, if i test our data and above one, i couldn't recreate it. Always it is coming in proper order (11,32,23). Nobody believes me that EQUALS is causing the issue:)
Joined: 07 Dec 2007 Posts: 2205 Location: San Jose
kbmk wrote:
Hi Bill,
We tested it so many times but we didn't notice any change. This code has been in production and running from May and issue was noticed only on August last week. Our business questions why it took so long to create an issue? I guess that's a valid question.
kbmk,
Run this JCL and pay attention to the 6th byte, which is in the ascending order. I am sorting on the first 3 bytes which is basically the word "key" for every record
Code:
//STEP0100 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTIN DD *
KEY0 A
KEY3 B
KEY4 C
KEY1 D
KEY2 E
KEY3 F
KEY4 G
KEY5 H
KEY6 I
KEY7 J
KEY0 K
KEY0 L
KEY0 M
KEY0 N
KEY0 O
KEY0 P
//SORTOUT DD SYSOUT=*
//SYSIN DD *
INREC OVERLAY=(23450:C'KBMK WANTS PROOF FOR EQUALS')
SORT FIELDS=(1,3,CH,A),NOEQUALS
//*
kbmk wrote:
Nobody believes me that EQUALS is causing the issue:)
This is one of those urban legends at your site that just won't die? The "true believers" fall into the category "don't bother me with the facts, my mind is made up".
I guess you have sorted on 1,4 rather than 1,3 (saying it based on fourth character in your output - it is sorted in ascending order). I tried that. Below is the output and still no difference.
Code:
Input EQUALS NOEQUALS
KEY0 A KEY0 A KEY0 A
KEY3 B KEY0 K KEY0 K
KEY4 C KEY0 L KEY0 L
KEY1 D KEY0 M KEY0 M
KEY2 E KEY0 N KEY0 N
KEY3 F KEY0 O KEY0 O
KEY4 G KEY0 P KEY0 P
KEY5 H KEY1 D KEY1 D
KEY6 I KEY2 E KEY2 E
KEY7 J KEY3 B KEY3 B
KEY0 K KEY3 F KEY3 F
KEY0 L KEY4 C KEY4 C
KEY0 M KEY4 G KEY4 G
KEY0 N KEY5 H KEY5 H
KEY0 O KEY6 I KEY6 I
KEY0 P KEY7 J KEY7 J
Joined: 07 Dec 2007 Posts: 2205 Location: San Jose
kbmk wrote:
I guess you have sorted on 1,4 rather than 1,3 (saying it based on fourth character in your output - it is sorted in ascending order). I tried that. Below is the output and still no difference.
Kbmk,
I have sorted only the first 3 bytes and trust me when I say that.
Code:
ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE751I 0 C5-K76982 C6-K90026 C7-K94453 C8-K94453 E9-K60824 C9-BASE E5-K80744
ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AN
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 13:14 ON WED SE
INREC OVERLAY=(23450:C'KBMK WANTS PROOF FOR EQUALS')
SORT FIELDS=(1,3,CH,A),NOEQUALS
ICE193I 0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
ICE088I 1 KOLUSUA .STEP0100. , INPUT LRECL = 80, BLKSIZE = 80, TYPE = FB
ICE093I 0 MAIN STORAGE = (MAX,6291456,6278238)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (6172656,6172656)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,
ICE128I 0 OPTIONS: SIZE=6291456,MAXLIM=1048576,MINLIM=450560,EQUALS=N,LIST=Y,ERE
ICE129I 0 OPTIONS: VIO=N,RESDNT=NONE,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITC
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMI
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y
ICE084I 0 BSAM ACCESS METHOD USED FOR SORTOUT
ICE084I 0 BSAM ACCESS METHOD USED FOR SORTIN
ICE750I 0 DC 500000 TC 0 CS DSVNN KSZ 3 VSZ 3
ICE752I 0 FSZ=6250 RC IGN=0 E AVG=23476 0 WSP=190571 C DYN=0 0
ICE751I 1 DE-K83743 D5-K91600 D9-K61787 E8-K94453
ICE090I 0 OUTPUT LRECL = 23476, BLKSIZE = 23476, TYPE = FB
ICE171I 0 SORTOUT LRECL OF 23476 IS DIFFERENT FROM SORTIN(NN) LRECL OF 80 - RC=0
ICE080I 0 IN MAIN STORAGE SORT
ICE055I 0 INSERT 0, DELETE 0
ICE054I 0 RECORDS - IN: 16, OUT: 16
ICE134I 0 NUMBER OF BYTES SORTED: 1280
ICE253I 0 RECORDS SORTED - PROCESSED: 16, EXPECTED: 6250
ICE191I 0 DFSORT SVC NOT AVAILABLE - PERFORMANCE MAY BE DEGRADED
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 0 , TRACKS USED: 0
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
Please run your job adding //SORTDIAG DD DUMMY to your sort step and send us the complete output to dfsort@us.ibm.com
Joined: 07 Dec 2007 Posts: 2205 Location: San Jose
kbmk wrote:
I am not sure why it is looking for DFMSGOUT. I also expect to see a difference in the output when I use NOEQUALS. But I do not see any difference.
SYSOUT DD identifies the DFSORT message data set. The default ddname is SYSOUT, but you can specify an alternate ddname for the message data set using the MSGDDN installation or run-time option. Always supply a DD statement for the message data set if a catalogued procedure is not used. (If you are invoking DFSORT from a COBOL program and are using the ddname SYSOUT for the message data set, the use of DISPLAY in your COBOL program can produce uncertain printing results.)
So your shop must have changed the defaults. Run this job to list your shop defaults. Please send the output of this job also to the same email address listed
Joined: 07 Dec 2007 Posts: 2205 Location: San Jose
kbmk,
I just checked the emails you have sent and unfortunately you are running DFSORT V1R10 which is out of service. We are not supposed to work on releases that are out of service. If your shop has an extended contract with IBM then you can open a PMR and we will look into it.
