IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

How to remove all of the duplicates
IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic Quick References
View previous topic :: View next topic  
Author Message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India
PostPosted: Wed May 05, 2010 5:46 pm
 Reply with quote
How can I remove the duplicates from a file. I know using sort with SUM FIELDS=NONE can be used to remove the duplicates but suppose if a file contains 5 duplicate records and i am using SORT utility with sum fields=none then it is removing 4 records and one record is left out of those five . I don't want any of those five records at all.
Back to top
View user's profile Send private message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India
PostPosted: Wed May 05, 2010 6:01 pm
 Reply with quote
RECFM is FB and LRECL=120, my Ouput file will also be having the same LRECL and RECFM.
Back to top
View user's profile Send private message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India
PostPosted: Wed May 05, 2010 6:14 pm
 Reply with quote
My client mainframe is using the DFSORT

Code:

ICE143I 0 BLOCKSET     COPY  TECHNIQUE SELECTED                                 
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AN
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R10 - 08:40 ON WED MA
            SORT     FIELDS=COPY                                               
ICE201I F RECORD TYPE IS F - DATA STARTS IN POSITION 1


This is the screenshot of the spool. Hope it helps now.
Back to top
View user's profile Send private message
Gnanas N

Active Member


Joined: 06 Sep 2007
Posts: 792
Location: Chennai, India
PostPosted: Wed May 05, 2010 6:17 pm
 Reply with quote
Hi,

Something like below?

Code:
//STEP0100 EXEC PGM=ICETOOL                         
//TOOLMSG  DD SYSOUT=*                             
//DFSMSG   DD SYSOUT=*                             
//T1       DD *                                     
12                                                 
12                                                 
13                                                 
15                                                 
13                                                 
99                                                 
//UNQ      DD SYSOUT=*                             
//TOOLIN   DD *                                     
  SELECT FROM(T1) TO(UNQ) ON(1,2,CH) NODUPS         
/*                                                 


Output:
Code:
********************************* Top of Data **********************************
15                                                                             
99                                                                             
******************************** Bottom of Data ********************************

And
NODUPS is equivalent to EQUAL(1) or LOWER(2).

Please let me know if any questions.
Back to top
View user's profile Send private message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India
PostPosted: Wed May 05, 2010 6:24 pm
 Reply with quote
Yeah... rite... Thanks a lot.. it is serving my purpose. icon_smile.gif
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales
PostPosted: Wed May 05, 2010 6:54 pm
 Reply with quote
As an aside, I noticed that SELECT performs a sort of the data.

If I wanted to retain the original data order I can use ICETOOL with multiple passes of the data, but could it be done more efficiently.

ICE201I G RECORD TYPE IS F

Code:
//IN1      DD DSN=&&TEMP01,DISP=SHR                     
//IN2      DD DSN=&&TEMP02,DISP=(,PASS,DELETE),         
//            SPACE=(TRK,(1,1)),RECFM=FB,LRECL=88       
//IN3      DD DSN=&&TEMP03,DISP=(,PASS,DELETE),         
//            SPACE=(TRK,(1,1)),RECFM=FB,LRECL=88       
//IN4      DD DSN=&&TEMP04,DISP=(,PASS,DELETE),         
//            SPACE=(TRK,(1,1)),RECFM=FB,LRECL=88       
//REPOUT   DD SYSOUT=*                                   
//TOOLIN   DD *                                         
 COPY FROM(IN1)   TO(IN2)    USING(CTL1)                 
 SELECT FROM(IN2) TO(IN3) ON(1,80,CH) NODUPS DISCARD(IN4)
 COPY FROM(IN3)   TO(REPOUT) USING(CTL2)                 
/*                                                       
//CTL1CNTL DD *                                         
  OUTREC BUILD=(1:1,80,81:SEQNUM,8,ZD)                   
/*                                                       
//CTL2CNTL DD *                                         
  SORT FIELDS=(81,8,ZD,A)                               
  OUTREC BUILD=(1:1,80)                                 
/*
Back to top
View user's profile Send private message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India
PostPosted: Wed May 05, 2010 7:13 pm
 Reply with quote
Thanks a lot all. I am using the solution suggested by expat. it will be keeping the order of the records same. Thanks
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose
PostPosted: Wed May 05, 2010 9:15 pm
 Reply with quote
Gagangarg wrote:
Thanks a lot all. I am using the solution suggested by expat. it will be keeping the order of the records same. Thanks


Gagangarg,

If you want to maintain the original order of records then use the following DFSORT/ICETOOL JCL. Assuming your input is FB recfm and 80 bytes length

Code:

//STEP0100 EXEC PGM=ICETOOL                               
//TOOLMSG  DD SYSOUT=*                                     
//DFSMSG   DD SYSOUT=*                                     
//IN       DD *                                           
99                                                         
12                                                         
12                                                         
13                                                         
15                                                         
13                                                         
//OUT      DD SYSOUT=*                                     
//T1       DD DSN=&&T1,DISP=(,PASS),SPACE=(CYL,(1,1),RLSE)
//TOOLIN   DD *                                           
  SELECT FROM(IN) TO(T1) ON(1,2,CH) NODUPS USING(CTL1)     
  SORT FROM(T1) USING(CTL2)                               
//CTL1CNTL DD *                                           
  INREC OVERLAY=(81:SEQNUM,8,ZD)                           
//CTL2CNTL DD *                                           
  SORT FIELDS=(81,8,CH,A)                                 
  OUTFIL FNAMES=OUT,BUILD=(1,80)                           
//*


expat,

Is there a reason as to why you have the DCB parameters hard coded in your job? SORT automatically calculates DCB from the input file or the control cards (inrec/outrec)
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales
PostPosted: Thu May 06, 2010 11:36 am
 Reply with quote
Skolusu wrote:
expat,
Is there a reason as to why you have the DCB parameters hard coded in your job? SORT automatically calculates DCB from the input file or the control cards (inrec/outrec)

Just force of habit.

I do not get too many opportunities to use much of the DFSORT functionality and just code it up like I would most jobs.

Thanks for the updated code. After seeing your code and after a quick search in my archive library it appears I did actually have an example of the SELECT statement which included the USING parameter

Just another sign of advancing years perhaps icon_lol.gif
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 

Similar Topics
Topic Forum Replies
No new posts Sortjoin and Search for a String and ... DFSORT/ICETOOL 1
No new posts Remove leading zeroes SYNCSORT 4
No new posts How to remove block of duplicates DFSORT/ICETOOL 8
This topic is locked: you cannot edit posts or make replies. Compare files with duplicates in one ... DFSORT/ICETOOL 11
No new posts To Remove spaces (which is in hex for... JCL & VSAM 10
Search our Forums:

Back to Top
IBMMainframes.com is not an official and/or affiliated with IBM® in anyway
Board Rules | FAQ | Downloads | Wiki | SiteMap | Contact Us