Step 1- File A with Lrecl 1500 and huge nnumber of records is Sorted using and it produces File B (unique records) and File C (Duplicate records in Xsum) Out of 42 milloin records only 1 million is duplicates.
SORT FIELDS=(1,18,CH,A)
SUM FIELDS=NONE,XSUM
Step 2 - Sorts the File B (unique records) produced in step 1 by
SORT FIELDS=(22,4,CH,A,26,21,CH,A,98,3,CH,A)
and produces File D . Each of these steps run for 2.5 hours .
My Question is , if we ombine these 2 sort steps into one (Doing the second sort along with Xsum like one given below) , will it provide savings in elapsed time ?
SORT FIELDS=(1,18,CH,A,22,4,CH,A,26,21,CH,A,98,3,CH,A)
SUM FIELDS=NONE,XSUM
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
Hello,
The first step eliminates duplicates on one field - positions 1-18.
The combined control statement would not eliminate these same duplicates. . . The number of "unique" records would increase (most likely) because of the additional fields.
How many processes use the sorted file of only unique records? If only 1 or a few, could duplicates simply be ignored then?
SORTOF2 contains only duplicates and need not be sorted . I verified that duplicates are required for Business. I am going to try the new Sort card and keep everyone posted.
Yes. I tested this . My uique records are sorted in (1,18,CH,A,22,4,CH,A,26,21,CH,A,98,3,CH,A) whereas it should be in FIELDS=(22,4,CH,A,26,21,CH,A,98,3,CH,A ) order .
Also duplicates should be in 1, 18 order but it is same order as above.
So that unique records are sorted in correct order. But is it possible to give a sort condition in OUTFIL FILES=02 so that duplicates are in (1, 18 , CH, A ) order ?
Joined: 17 Oct 2006 Posts: 2481 Location: @my desk
Quote:
So that unique records are sorted in correct order
Yes. It'll be sorted in correct order. But the records may not be unique now
OUTREC OVERLAY=(1501:SEQNUM,8,ZD,RESTART=(1,18)). The RESTART parameter here would work combining with the SORT FIELDS=(1,18,...) so that INCLUDE given in your OUTFIL FILES=01 extracts only the first record out of each key occurring at pos 1-18. Now that you have removed pos 1-18 from the sort fields, it may not be extracting 'unique' records.
My uique records are sorted in (1,18,CH,A,22,4,CH,A,26,21,CH,A,98,3,CH,A) whereas it should be in FIELDS=(22,4,CH,A,26,21,CH,A,98,3,CH,A ) order .
Sorry, I read through the requirements too fast and missed the detail regarding the different sort order for the second output file. I was focusing on your question regarding savings in elapsed time.
