Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

How to remove all of the duplicates

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL
View previous topic :: :: View next topic  
Author Message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India

PostPosted: Wed May 05, 2010 5:46 pm    Post subject: How to remove all of the duplicates
Reply with quote

How can I remove the duplicates from a file. I know using sort with SUM FIELDS=NONE can be used to remove the duplicates but suppose if a file contains 5 duplicate records and i am using SORT utility with sum fields=none then it is removing 4 records and one record is left out of those five . I don't want any of those five records at all.
Back to top
View user's profile Send private message

GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India

PostPosted: Wed May 05, 2010 6:01 pm    Post subject:
Reply with quote

RECFM is FB and LRECL=120, my Ouput file will also be having the same LRECL and RECFM.
Back to top
View user's profile Send private message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India

PostPosted: Wed May 05, 2010 6:14 pm    Post subject:
Reply with quote

My client mainframe is using the DFSORT

Code:

ICE143I 0 BLOCKSET     COPY  TECHNIQUE SELECTED                                 
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AN
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R10 - 08:40 ON WED MA
            SORT     FIELDS=COPY                                               
ICE201I F RECORD TYPE IS F - DATA STARTS IN POSITION 1


This is the screenshot of the spool. Hope it helps now.
Back to top
View user's profile Send private message
Gnanas N

Active Member


Joined: 06 Sep 2007
Posts: 788
Location: Chennai, India

PostPosted: Wed May 05, 2010 6:17 pm    Post subject: Reply to: How to remove all of the duplicates
Reply with quote

Hi,

Something like below?

Code:
//STEP0100 EXEC PGM=ICETOOL                         
//TOOLMSG  DD SYSOUT=*                             
//DFSMSG   DD SYSOUT=*                             
//T1       DD *                                     
12                                                 
12                                                 
13                                                 
15                                                 
13                                                 
99                                                 
//UNQ      DD SYSOUT=*                             
//TOOLIN   DD *                                     
  SELECT FROM(T1) TO(UNQ) ON(1,2,CH) NODUPS         
/*                                                 


Output:
Code:
********************************* Top of Data **********************************
15                                                                             
99                                                                             
******************************** Bottom of Data ********************************

And
NODUPS is equivalent to EQUAL(1) or LOWER(2).

Please let me know if any questions.
Back to top
View user's profile Send private message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India

PostPosted: Wed May 05, 2010 6:24 pm    Post subject:
Reply with quote

Yeah... rite... Thanks a lot.. it is serving my purpose. icon_smile.gif
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Wed May 05, 2010 6:54 pm    Post subject:
Reply with quote

As an aside, I noticed that SELECT performs a sort of the data.

If I wanted to retain the original data order I can use ICETOOL with multiple passes of the data, but could it be done more efficiently.

ICE201I G RECORD TYPE IS F

Code:
//IN1      DD DSN=&&TEMP01,DISP=SHR                     
//IN2      DD DSN=&&TEMP02,DISP=(,PASS,DELETE),         
//            SPACE=(TRK,(1,1)),RECFM=FB,LRECL=88       
//IN3      DD DSN=&&TEMP03,DISP=(,PASS,DELETE),         
//            SPACE=(TRK,(1,1)),RECFM=FB,LRECL=88       
//IN4      DD DSN=&&TEMP04,DISP=(,PASS,DELETE),         
//            SPACE=(TRK,(1,1)),RECFM=FB,LRECL=88       
//REPOUT   DD SYSOUT=*                                   
//TOOLIN   DD *                                         
 COPY FROM(IN1)   TO(IN2)    USING(CTL1)                 
 SELECT FROM(IN2) TO(IN3) ON(1,80,CH) NODUPS DISCARD(IN4)
 COPY FROM(IN3)   TO(REPOUT) USING(CTL2)                 
/*                                                       
//CTL1CNTL DD *                                         
  OUTREC BUILD=(1:1,80,81:SEQNUM,8,ZD)                   
/*                                                       
//CTL2CNTL DD *                                         
  SORT FIELDS=(81,8,ZD,A)                               
  OUTREC BUILD=(1:1,80)                                 
/*
Back to top
View user's profile Send private message
GaganGarg

Active User


Joined: 31 Mar 2010
Posts: 134
Location: India

PostPosted: Wed May 05, 2010 7:13 pm    Post subject:
Reply with quote

Thanks a lot all. I am using the solution suggested by expat. it will be keeping the order of the records same. Thanks
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose

PostPosted: Wed May 05, 2010 9:15 pm    Post subject: Reply to: How to remove all of the duplicates
Reply with quote

Gagangarg wrote:
Thanks a lot all. I am using the solution suggested by expat. it will be keeping the order of the records same. Thanks


Gagangarg,

If you want to maintain the original order of records then use the following DFSORT/ICETOOL JCL. Assuming your input is FB recfm and 80 bytes length

Code:

//STEP0100 EXEC PGM=ICETOOL                               
//TOOLMSG  DD SYSOUT=*                                     
//DFSMSG   DD SYSOUT=*                                     
//IN       DD *                                           
99                                                         
12                                                         
12                                                         
13                                                         
15                                                         
13                                                         
//OUT      DD SYSOUT=*                                     
//T1       DD DSN=&&T1,DISP=(,PASS),SPACE=(CYL,(1,1),RLSE)
//TOOLIN   DD *                                           
  SELECT FROM(IN) TO(T1) ON(1,2,CH) NODUPS USING(CTL1)     
  SORT FROM(T1) USING(CTL2)                               
//CTL1CNTL DD *                                           
  INREC OVERLAY=(81:SEQNUM,8,ZD)                           
//CTL2CNTL DD *                                           
  SORT FIELDS=(81,8,CH,A)                                 
  OUTFIL FNAMES=OUT,BUILD=(1,80)                           
//*


expat,

Is there a reason as to why you have the DCB parameters hard coded in your job? SORT automatically calculates DCB from the input file or the control cards (inrec/outrec)
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Thu May 06, 2010 11:36 am    Post subject: Re: Reply to: How to remove all of the duplicates
Reply with quote

Skolusu wrote:
expat,
Is there a reason as to why you have the DCB parameters hard coded in your job? SORT automatically calculates DCB from the input file or the control cards (inrec/outrec)

Just force of habit.

I do not get too many opportunities to use much of the DFSORT functionality and just code it up like I would most jobs.

Thanks for the updated code. After seeing your code and after a quick search in my archive library it appears I did actually have an example of the SELECT statement which included the USING parameter

Just another sign of advancing years perhaps icon_lol.gif
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Remove junk values in a file and rite... KP1125 DFSORT/ICETOOL 2 Wed Jan 25, 2017 9:58 pm
No new posts Sort Card to Remove Duplicate records... raj4neo SYNCSORT 2 Wed Jan 25, 2017 4:44 am
No new posts Removing Duplicates based on certain ... chandracdac DFSORT/ICETOOL 8 Fri Dec 09, 2016 4:40 am
No new posts Updating the counters after eliminati... PANDU1 DFSORT/ICETOOL 12 Mon Nov 21, 2016 9:47 am
No new posts Remove Special Characters from Mainfr... Rodger Zhang All Other Mainframe Topics 6 Wed Jul 06, 2016 1:12 am


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us