I made a search effort but couldn't find the following as a topic.
I've wanted to find duplicates in an already sorted file but couldn't see anything in the manuals that says you can do it without re-sorting the file.
Hence I've had to code
sort fields=(...)
sum fields=none
in order that the duplicates be counted and removed. Of course, what is happening is that the SUM operation needs to be able to recognise duplicates, and the only way to do this is by the SORT statement, telling of the control fields. If there's a better way then I'm all ears!
By the way, regarding searching, the search box is available in the initial forum point but is unavailable after that. Hence, when a search brings back thousands of records you cannot do a further automatic search through the sub-set. You might say that a better search string should be given, but there is often going to be an occasion when searching returns hundreds or thousands of records. Thus, searching should be available for sub-sets.
Joined: 07 Dec 2007 Posts: 2205 Location: San Jose
Jim Alton,
You can use SELECT operator with OPTION COPY which will give you the desired results without performing a sort
Code:
//STEP0100 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN DD *
ABC MY #1 DUP RECORD
ABC MY #2 DUP RECORD
ABC MY #3 DUP RECORD
ABC MY #4 DUP RECORD
DEF MY #1 UNIQ RECORD
//OUT DD SYSOUT=*
//TOOLIN DD *
SELECT FROM(IN) TO(OUT) ON(1,3,CH) FIRST USING(CTL1)
//CTL1CNTL DD *
OPTION COPY
//*
ICE055I 0 INSERT 0, DELETE 0
ICE054I 0 RECORDS - IN: 119922, OUT: 119922
ICE227I 0 SORTOUT : DELETED = 119922, REPORT = 71039, DATA = 0
ICE228I 0 SORTOUT : TOTAL IN = 119922, TOTAL OUT = 71039
ICE174I 0 NO DATA RECORDS FOR AN OUTFIL DATA SET - RC=0
So there were 119922 input records of which 71039 are unique and 48883 (119922 - 71039) are duplicates that were deleted, but among the 71039 are 'originals' of the deleted duplicates. These figures tally with many runs of this step with the old method I used. Then I tried ICETOOL.
You can see that the JCL is just about the same, but the output counts vary.
Here are salient output messages, and note that I not only tried FIRST but also FIRST(1),LAST, ALLDUPS, NODUPS, but in every case the counts, except for the total number of records, don't match with what I've had before.
Code:
ICE000I 0 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 09:37 ON THU APR 04, 2013 -
OPTION COPY
ICE146I 0 END OF STATEMENTS FROM CTL1CNTL - PARAMETER LIST STATEMENTS FOLLOW
DEBUG NOABEND,ESTAE
OPTION MSGDDN=DFSMSG,LIST,MSGPRT=ALL,RESINV=0,SORTDD=CTL1,SORTIN=IN,SORTOUT=OUT,DYNALLOC,SZERO,EQUALS,NOVLSHRT,LOCALE=NONE,NOCHECK
SORT FIELDS=(157,7,CH,A)
MODS E35=(ICE35DU,12288)
Joined: 07 Dec 2007 Posts: 2205 Location: San Jose
Jim Alton,
Sections is using BINARY comparison and select is using Character comparison. DFSORT sorts BI and CH data the same unless there's an option in effect that affects the collating sequence of the CH data like LOCALE or CHALT.
How about changing SELECT also to BI and see if you are still getting different results?
If you are still getting different results then please add the following to your job and re-run the job and send us the complete sysout to dfsort@us.ibm.com.
In the following I've provided the step jcl and the sort messages. Note that in the ICETOOL SELECT I've used BI instead of CH, as suggested, but it has made no difference; but then it shouldn't since we are not doing a sort in this step but a copy of an already sorted file, and, presumably, ICETOOL is simply making a comparison on the control field to see if the current record's control field is equal to the previous record's control field, and for a simple equals comparison the collating sequence would be immaterial. Note that I've copied and pasted the following from a 3270, 80 byte screen so some of the messages are truncated but I'm hoping that it won't matter.
SELECT FROM(IN) TO(OUT) ON(157,7,BI) FIRST USING(CTL1)
ICE606I 0 DFSORT CALL 0001 FOR SORT FROM IN TO OUT USING CTL1CNTL CO
ICE628I 0 RECORD COUNT: 000000000119922
ICE638I 0 NUMBER OF RECORDS RESULTING FROM CRITERIA: 000000000090800
ICE602I 0 OPERATION RETURN CODE: 00
I followed Kolusu's advice and included SORTDIAG in each step, as can be seen by the JCL I included in my last message; because of this, I am puzzled by the meaning of your question: "Did you follow Kolusu's advice with the SORTDIAG and ship the output of to DFSORT?". There is extra information in the output messages that you should recognise, presumably provided because of the SORTDIAG DD statement.
Regarding the truncation of the messages at 80 bytes, if you really can't determine much from what I've already supplied then I'll try to supply the truncated parts as well; however, the method you suggest, XDC, I don't recognise - is it a standard function of an IBM setup?
Joined: 10 May 2007 Posts: 2455 Location: Hampshire, UK
XDC is an SDSF function to output data from the spool to a dataset. What you seem to be missing is that Kolusu asked you to send the sysout to him by e-mail - not post it here.
Thanks, Nic - I've never exported SDSF output to a dataset, but I'll see if I can do it on Monday. I hope it's an option that is available to me - i.e. that it's a standard function within SDSF that isn't sometimes made unavailable.
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
Yes, it was the e-mailing to the DFSORT people I was asking about, sorry to not be clear, Jim.
The SYSOUT to the dataset is another way (requiring a re-run, of course). If you have the output sitting in the queue, ? the job, XDC against the relevant sysouts for the relevant steps, and you should be prompted from there. If that doesn't work, then yes, it's probably been chopped. Perhaps look at "SDSF in batch"?
Thanks, Mistah Kurtz - the most obvious solution had to come from someone with second thoughts!
Also, thanks for the link about sending SDSF output to a dataset (XDC). After being informed about XDC I imagined that there would be an SDSF line command or separate panel, but I can see from that link that it would involve coding a REXX exec, referring to SDSF panels, none of which I've done, though I would probably have given it a go, except you've suggested a simpler - though less interesting - solution!
Joined: 28 Jan 2012 Posts: 316 Location: Room: TREE(3). Hilbert's Hotel
Hi..XDC can be used as line command as below:
Code:
SDSF JOB DATA SET DISPLAY - JOB XXXXXXXX (JOB00001) LINE 1-11 (11)
COMMAND INPUT ===> SCROLL ===> CSR
NP DDNAME StepName ProcStep DSID Owner C Dest Rec-Cnt Page
JESMSGLG JES2 2 XXXXXXX H LOCAL 35
JESJCL JES2 3 XXXXXXX H LOCAL 64
JESYSMSG JES2 4 XXXXXXX H LOCAL 154
xdc SYSOUT STEP001 107 XXXXXXX H LOCAL 104
And then you get the next Panel:
Code:
SDSF Open Print Data Set
COMMAND INPUT ===> SCROLL ===> CSR
Data set name ===> 'XXX.YYYY.ZZZ'
Member to use ===>
Disposition ===> NEW (OLD, NEW, SHR, MOD)
If the data set is to be created, specify the following.
Volume serial will be used to locate existing data sets if specified.
Management class ===> (Blank for default management class)
Storage class ===> (Blank for default storage class)
Volume serial ===> (Blank for authorized default volume)
Device type ===> (Generic unit or device address)
Data class ===> (Blank for default data class)
Space units ===> TRKS (BLKS, TRKS, CYLS, BY, KB, or MB)
Primary quantity ===> 5 (In above units)
Secondary quantity ===> 10 (In above units)
Directory blocks ===> 0 (Zero for sequential data set)
Record format ===> FB
Record length ===> 133
Sorry..I was not very clear earlier..my apologies !!!
Thanks for the demonstration Mistah Kurtz - as I said earlier, I hope XDC is a standard function that is available to me, and I'll be able to determine that tomorrow, Monday.
By the way, how are you copying your mainframe screen into the forum's reply box? You can see from my earlier replies that copying and pasting from the mainframe screen results in text being imported, but you seem to have saved your screen as an image and pasted that image via the image option of BBCode - correct?
Joined: 28 Jan 2012 Posts: 316 Location: Room: TREE(3). Hilbert's Hotel
Jim Alton wrote:
In the following I've provided the step jcl and the sort messages. Note that in the ICETOOL SELECT I've used BI instead of CH, as suggested, but it has made no difference; but then it shouldn't since we are not doing a sort in this step but a copy of an already sorted file, and, presumably, ICETOOL is simply making a comparison on the control field to see if the current record's control field is equal to the previous record's control field, and for a simple equals comparison the collating sequence would be immaterial. Note that I've copied and pasted the following from a 3270, 80 byte screen so some of the messages are truncated but I'm hoping that it won't matter.
ICE650I 0 VISIT http://www.ibm.com/storage/dfsort FOR ICETOOL PAPERS, EXAMPLES A
ICE632I 0 SOURCE FOR ICETOOL STATEMENTS: TOOLIN
ICE630I 0 MODE IN EFFECT: STOP
SELECT FROM(IN) TO(OUT) ON(157,7,BI) FIRST USING(CTL1)
ICE606I 0 DFSORT CALL 0001 FOR SORT FROM IN TO OUT USING CTL1CNTL CO
ICE628I 0 RECORD COUNT: 000000000119922
ICE638I 0 NUMBER OF RECORDS RESULTING FROM CRITERIA: 000000000090800
ICE602I 0 OPERATION RETURN CODE: 00
Thanks for the advice on 'Code' - I see how it works.
Also, XDC worked, and I transferred the datasets to text files and mailed them off to Skolusu a few minutes ago.
Joined: 07 Dec 2007 Posts: 2205 Location: San Jose
Jim Alton wrote:
Thanks for the advice on 'Code' - I see how it works.
Also, XDC worked, and I transferred the datasets to text files and mailed them off to Skolusu a few minutes ago.
I received the documentation and Sent you an email with follow up questions. Please respond to them.