IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Sort utility step to eliminate duplicate records
IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic Quick References
View previous topic :: View next topic  
Author Message
Nagendran

New User


Joined: 24 Jan 2007
Posts: 89
Location: USA
PostPosted: Mon Sep 24, 2007 12:46 pm
 Reply with quote
Hi ,

I am using below step to eliminate duplicate records from the file , but the output file is having the duplicate records.

can any body help with below?

Code:

//STEP03  EXEC PGM=SORT,COND=(0,NE)               
//SORTIN  DD DSN=TEST.FILE2,               
//        DISP=SHR                                 
//SORTOUT DD DSN=TEST.FILE3,               
//        DISP=(MOD,CATLG,CATLG),                 
//        SPACE=(CYL,(300,300),RLSE),             
//        UNIT=TESTDA,                             
//        DCB=(LRECL=80,RECFM=FB,BLKSIZE=0)       
//SORTWK01 DD UNIT=SYSDA,DISP=(NEW,CATLG,CATLG),   
//         SPACE=(TRK,(30,30),RLSE)               
//SYSOUT DD SYSOUT=*                               
//SYSIN   DD *                                     
  SORT FIELDS=COPY                                 
  SUM FIELDS=NONE                                 
/*       


regards,
Nagu
Back to top
View user's profile Send private message
guptae

Moderator


Joined: 14 Oct 2005
Posts: 1208
Location: Bangalore,India
PostPosted: Mon Sep 24, 2007 12:52 pm
 Reply with quote
Hi Nagendran,

If you want to remove duplicate record on the basis of key then please provide the follwing sort card

Code:

  SORT FIELDS=(1,80,CH,A)
  SUM FIELDS=NONE


here i assume that u wnat to remove duplicate on the bassis of thre whole record
But if u have any specific keys then u can change the record accordingly.please read the manual on sort for more clarification
Back to top
View user's profile Send private message
Nagendran

New User


Joined: 24 Jan 2007
Posts: 89
Location: USA
PostPosted: Mon Sep 24, 2007 12:57 pm
 Reply with quote
hi,


I dont want to sort... I want to eliminate only the duplicates.

thanks,
nagu
Back to top
View user's profile Send private message
murmohk1

Senior Member


Joined: 29 Jun 2006
Posts: 1436
Location: Bangalore,India
PostPosted: Mon Sep 24, 2007 1:01 pm
 Reply with quote
Nagendran,

Quote:
I dont want to sort... I want to eliminate only the duplicates.

You are supposed to sort for elimination dups.
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10873
Location: italy
PostPosted: Mon Sep 24, 2007 1:01 pm
 Reply with quote
Quote:


I dont want to sort... I want to eliminate only the duplicates.


... by grace, will You tell us how to do it without sorting

regards

e.s
Back to top
View user's profile Send private message
guptae

Moderator


Joined: 14 Oct 2005
Posts: 1208
Location: Bangalore,India
PostPosted: Mon Sep 24, 2007 1:05 pm
 Reply with quote
Hi There,

Use the following jcl

Code:
//STEP001  EXEC PGM=SORT
//SORTLIB  DD DSN=SYS1.SORTLIB,DISP=SHR                                 
//SORTWK01 DD SPACE=(CYL,(100)),UNIT=DISK                               
//SYSIN    DD  *
   INREC FIELDS=(1,80,SEQNUM,8,ZD)
   SORT FIELDS=(1,80,CH,A)
   SUM FIELDS=NONE
/*
//SORTIN   DD  DSN=TBH2.SUM.SORTIN.DATA,DISP=SHR
//SORTOUT  DD  DSN=TBH2.SUM.SORTOUT.DATA,
//             UNIT=DISK,SPACE=(TRK,(1,1),RLSE),DISP=(,CATLG,DELETE),
//             DCB=(LRECL=88,BLKSIZE=880,RECFM=FB)
//SYSOUT   DD  SYSOUT=*
//SYSOUT   DD SYSOUT=*                                                 
//STATOUT DD SYSOUT=*                                                   
//SYSUDUMP DD SYSOUT=*                                               
//*
//STEP002  EXEC PGM=SORT
//SORTLIB  DD DSN=SYS1.SORTLIB,DISP=SHR                                 
//SORTWK01 DD SPACE=(CYL,(100)),UNIT=DISK                               
//SYSIN    DD  *
   SORT FIELDS=(80,8,CH,A)
   OUTREC FIELDS=(1:1,80)
/*
//SORTIN   DD  DSN=TBH2.SUM.SORTOUT.DATA,DISP=SHR
//SORTOUT  DD  DSN=TBH2.SUM.SORTOUT1.DATA,
//             UNIT=DISK,SPACE=(TRK,(1,1),RLSE),DISP=(,CATLG,DELETE),
//             DCB=(LRECL=80,BLKSIZE=800,RECFM=FB)
//SYSOUT   DD  SYSOUT=*
//SYSOUT   DD SYSOUT=*                                                 
//STATOUT DD SYSOUT=*                                                   
//SYSUDUMP DD SYSOUT=*


Here i am assuming that u want to remove the duplicate on the basis of entire record .
First sort card will add an sequence number starting at 81 th byte & sort the input file on the basis of 1 to 80 th byte & remove the duplicate
Second sort will put the records in there orignal order.
Back to top
View user's profile Send private message
Frank Yaeger

DFSORT Developer


Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA
PostPosted: Mon Sep 24, 2007 8:51 pm
 Reply with quote
Nagu,

SUM FIELDS=NONE requires the use of a sort key to identify the duplicate values. That sort key can be the entire record, but the sort key still must be specified to use SUM FIELDS=NONE.

If your records are not already in sorted order, but you want to remove the duplicates and keep the records in their original order, you can use a DFSORT/ICETOOL job like this:

Code:

//S1    EXEC  PGM=ICETOOL
//TOOLMSG   DD  SYSOUT=*
//DFSMSG   DD  SYSOUT=*
//IN DD *
CCCCCCCCCCCCCCCCCCCC
AAAAAAAAAAAAAAAAAAAA
DDDDDDDDDDDDDDDDDDDD
AAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBB
BBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCC
AAAAAAAAAAAAAAAAAAAA
/*
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//OUT DD SYSOUT=*
//TOOLIN   DD    *
SELECT FROM(IN) TO(T1) ON(1,80,CH) FIRST USING(CTL1)
SORT FROM(T1) TO(OUT) USING(CTL2)
/*
//CTL1CNTL DD *
  INREC OVERLAY=(81:SEQNUM,8,ZD)
//CTL2CNTL DD *
  SORT FIELDS=(81,8,ZD,A)
  OUTREC BUILD=(1,80)
/*


OUT would have:

Code:

CCCCCCCCCCCCCCCCCCCC 
AAAAAAAAAAAAAAAAAAAA 
DDDDDDDDDDDDDDDDDDDD 
BBBBBBBBBBBBBBBBBBBB 


However, if your records are already in sorted order, you can avoid sorting them by using a MERGE or a COPY.

Here's an example of a DFSORT merge job:

Code:

//S2    EXEC  PGM=ICEMAN
//SYSOUT    DD  SYSOUT=*
//SORTIN01 DD *
AAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBB
BBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDD
/*
//SORTOUT DD SYSOUT=*
//SYSIN    DD    *
  OPTION EQUALS
  MERGE FIELDS=(1,80,CH,A)
  SUM FIELDS=NONE
/*


SORTOUT would have:

Code:

AAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDD


Here's an example of a DFSORT COPY job:

Code:

//S3    EXEC  PGM=ICEMAN
//SYSOUT    DD  SYSOUT=*
//SORTIN DD *
AAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBB
BBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDD
/*
//SORTOUT DD SYSOUT=*
//SYSIN    DD    *
  OPTION COPY
  OUTFIL REMOVECC,NODETAIL,
    SECTIONS=(1,80,
      TRAILER3=(1,80))
/*


SORTOUT would have:

Code:

AAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDD


In the future, please show an example of your input records and expected output records so we know what you are trying to do.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 

Similar Topics
Topic Forum Replies
No new posts Compare 2 files and retrive records f... DFSORT/ICETOOL 3
No new posts Compare 2 files(F1 & F2) and writ... JCL & VSAM 8
No new posts Need to set RC4 through JCL SORT DFSORT/ICETOOL 5
No new posts How to split large record length file... DFSORT/ICETOOL 10
No new posts Duplicate transid's declared using CEDA CICS 3
Search our Forums:

Back to Top
IBMMainframes.com is not an official and/or affiliated with IBM® in anyway
Board Rules | FAQ | Downloads | Wiki | SiteMap | Contact Us