Portal | Manuals | References | Downloads | Info | Programs | JCLs | Mainframe wiki | Quick Ref
IBM Mainframe Forum Index
 
Register
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Profile Log in to check your private messages Log in
 
Removing duplicates without Sorting in JCL
Goto page Previous  1, 2
 
Post new topic   This topic is locked: you cannot edit posts or make replies.    IBMMAINFRAMES.com Support Forums -> JCL & VSAM
View previous topic :: :: View next topic  
Author Message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Mon Apr 16, 2012 10:59 am    Post subject:
Reply with quote

Hi,

SYNCTOOL (alias ICETOOL) is part of SYNCSORT.


Gerry
Back to top
View user's profile Send private message

vishalbshah

New User


Joined: 01 Dec 2006
Posts: 61
Location: Pune

PostPosted: Mon Apr 16, 2012 11:12 am    Post subject:
Reply with quote

Hi Gerry,

I tried your suggetion but the same is not resulting in the expected outout.

Code:


//TCSVBSTS JOB 0000,'ICETOOL',CLASS=7,MSGCLASS=U,   
//    NOTIFY=&SYSUID,MSGLEVEL=(1,1)                 
//* $ACFJ219 ACF2 ACTIVE I003                       
//SORT10   EXEC PGM=ICETOOL                         
//IN       DD DSN=TCS.TEST.SORT.VISHAL,DISP=SHR     
//OUT      DD DSN=TCS.TEST.SORT.VISHAL1,DISP=SHR   
//TOOLMSG  DD   SYSOUT=*                           
//DFSMSG   DD   SYSOUT=*                           
//SORTWK01 DD   SPACE=(TRK,(1,1))                   
//TOOLIN   DD *                                     
SELECT FROM(IN) TO(OUT) ON(1,783,CH) FIRST         
//CTL1CNTL DD *                                     
SORT FIELDS=COPY                                   
/*                                                 



Output I am getting is

Code:

CDU MATCHING ENGINE REPORT FOR PROJECT CPP   |DATE:2012-04-12|FULL / PAR
NUMBER OF MATCHES RETURNED FOR BASE DATA :   |000011                   
ROWID|CPP PRODUCT ID|CANDIDATE CIN|CANDIDATE SIGN|CANDIDATE SCORE|FIRST
00001|CPP000562712  |0800000000   |+             |0100           |HAYLEY
00001|CPP000562712  |0800000017   |+             |0041           |MATHEW
00001|CPP000562712  |0800000018   |+             |0041           |HILLAR
00001|CPP000562712  |0800000019   |+             |0041           |HALE 
00001|CPP000562712  |0800000020   |+             |0041           |ADRIAN
00004|CPP000562752  |0800000000   |+             |0055           |HAYLEY
00004|CPP000562752  |0800000017   |+             |0041           |MATHEW
00004|CPP000562752  |0800000018   |+             |0041           |HILLAR
00004|CPP000562752  |0800000019   |+             |0041           |HALE 
00004|CPP000562752  |0800000020   |+             |0041           |ADRIAN
00005|CPP000562772  |0800000004   |+             |0055           |PAULIN


As you can see the trailer is now moved to header
it has removed the duplicates but the requirement is not to change the headers and trailer ordering.
Back to top
View user's profile Send private message
vishalbshah

New User


Joined: 01 Dec 2006
Posts: 61
Location: Pune

PostPosted: Mon Apr 16, 2012 11:16 am    Post subject:
Reply with quote

Hi Bill,

I have tried the alterantive which you suggested :

Code:


//TCSVBSTS JOB 0000,'SORT1',CLASS=7,MSGCLASS=U,                     
//    NOTIFY=&SYSUID,MSGLEVEL=(1,1)                                 
//* $ACFJ219 ACF2 ACTIVE I003                                       
//SORT10   EXEC PGM=SORT                                           
//SORTIN   DD DSN=TCS.TEST.SORT.VISHAL,DISP=SHR                     
//SORTOUT  DD DSN=TCS.TEST.SORT.VISHAL1,DISP=SHR                   
//SYSOUT   DD   SYSOUT=*                                           
//SYSOUZ   DD   SYSOUT=*                                           
//SORTWK01 DD   SPACE=(TRK,(1,1))                                   
//SYSIN    DD *                                                     
  OPTION COPY                                                       
  INREC IFTHEN=(WHEN=INIT,                                         
                OVERLAY=(784:SEQNUM,3,ZD,784,3,ZD,MOD,+2,EDIT=(T)))
  OUTREC IFTHEN=(WHEN=GROUP,BEGIN=(787,1,CH,EQ,C'1'),RECORDS=2,     
                PUSH=(788:1,4)),                                   
         IFTHEN=(WHEN=GROUP,BEGIN=(787,1,CH,EQ,C'0'),RECORDS=2,     
                PUSH=(792:1,4))                                     
  OUTFIL OMIT=(788,4,CH,EQ,792,4,CH),BUILD=(1,783)                 
/*                                                                 


but the result is not correct, seems to be only the matched detail record with headers and trailer appeared in the result.

Received Output:
Code:

CDU MATCHING ENGINE REPORT FOR PROJECT CPP   |DATE:2012-04-12|FULL / PAR
ROWID|CPP PRODUCT ID|CANDIDATE CIN|CANDIDATE SIGN|CANDIDATE SCORE|FIRST
00001|CPP000562712  |0800000000   |+             |0100           |HAYLEY
NUMBER OF MATCHES RETURNED FOR BASE DATA :   |000011                   


I am trying to change some conditions here ... please, let me knwo you can spot anything thing in this...
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Mon Apr 16, 2012 11:39 am    Post subject:
Reply with quote

Hi,

my mistake, my cut and paste was incorrect, it should be
Code:
SELECT FROM(IN) TO(OUT) ON(1,783,CH) FIRST USING(CTL1) 




Gerry
Back to top
View user's profile Send private message
vishalbshah

New User


Joined: 01 Dec 2006
Posts: 61
Location: Pune

PostPosted: Mon Apr 16, 2012 11:53 am    Post subject:
Reply with quote

Yes it works!

Thanks Gerry

Appreciate your help.

I have got the desired output.

so this card says move records from input to output as it is and when ever the records (1,783) are same in consecutive records just copy such first record.

Please, correct my understanding.
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Mon Apr 16, 2012 11:59 am    Post subject:
Reply with quote

Hi,

yes, it selects the FIRST occurrence of duplicate records without sorting

ie. SORT FIELDS=COPY


Gerry
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7312

PostPosted: Mon Apr 16, 2012 12:02 pm    Post subject: Reply to: Removing duplicates without Sorting in JCL
Reply with quote

Your file is not actually in "collating" sequence, strictly, as seen by the trailer moving its position.

Check whether select enjoys having a sorted file. If it does, test very carefully with Gerry's solution.

The problem with the way that you have applied my second solution is that you haven't specified the full extend of the record areas for matching.

With that one you have three record areas, all the same length, all as one record. You need to compare the second and third record areas, for the entire length of your record. I suspect you can only do this in 256-byte "chunks", hence my reference to 1,256 257,256 etc, where you have to offset the starting points for the two additional record areas.

So something likse 784,256,etc with each area matching against the third record area with AND joining each togeteher

a=a1,and,b=b1,and,c=c1,and,d=d1

where a,b,c,d are the chunks from the first extension (first PUSH) and a1,b1,c1,d1 are from the second.

EDIT: I see you are OK now.
Back to top
View user's profile Send private message
vishalbshah

New User


Joined: 01 Dec 2006
Posts: 61
Location: Pune

PostPosted: Mon Apr 16, 2012 12:07 pm    Post subject: Reply to: Removing duplicates without Sorting in JCL
Reply with quote

Hi Gerry,

Where can I see different options of ICETOOL?

I mean the one which we used i.e. "FIRST" works fine for now...

I want to see which other options are available with this.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7312

PostPosted: Mon Apr 16, 2012 12:11 pm    Post subject: Reply to: Removing duplicates without Sorting in JCL
Reply with quote

In your Syncsort documentation, of course.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7312

PostPosted: Mon Apr 16, 2012 11:32 pm    Post subject: Re: Reply to: Removing duplicates without Sorting in JCL
Reply with quote

Bill Woodger wrote:
In your Syncsort documentation, of course.


Or maybe you'd better contact Syncsort if you can't find it documented... let us know, please, how you get on.
Back to top
View user's profile Send private message
nagaraj_bez

New User


Joined: 09 Jun 2005
Posts: 10

PostPosted: Tue Apr 17, 2012 1:25 pm    Post subject:
Reply with quote

Hi Vishalbshah,

find below steps for removing duplicates(irrespective duplicate record position - contigious or not) with out sorting the file. Let me know the status once u tried this.

Used recod length as 80
&input = input file
&output2 = required output file

Code:
//COPY1  EXEC  PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTIN   DD  DSN=&INput ,
//             DISP=SHR
//SORTOUT  DD  DSN=&OUTPUT1,
//             DISP=(NEW,CATLG,DELETE),
//             RECFM=FB,LRECL=90,
//             SPACE=(TRK,(10,5),RLSE)
//SYSPRINT DD  SYSOUT=*
//SYSIN    DD  *
 INREC OVERLAY=(81:SEQNUM,8,ZD)
 SORT FIELDS=COPY
sum fields = none
 OUTREC BUILD=(1:81,8,9:1,80)
/*
//COPY2  EXEC  PGM=SORT
//SORTIN   DD  DSN=&OUTPUT1,
//             DISP=SHR
//SORTOUT  DD  DSN=&OUTPUT2,
//             DISP=(NEW,CATLG,DELETE),
//             RECFM=FB,LRECL=99,
//             SPACE=(TRK,(10,5),RLSE)
//SYSPRINT DD  SYSOUT=*
//SYSOUT   DD  SYSOUT=*
//SYSIN    DD  *
 SORT FIELDS=(1,8,CH,A)
 OUTREC BUILD=(1:1,89,X,91:SEQNUM,8,ZD,START=1,INCR=1,RESTART=(9,80))
/*
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7312

PostPosted: Tue Apr 17, 2012 1:38 pm    Post subject: Reply to: Removing duplicates without Sorting in JCL
Reply with quote

TS already has a solution which actually runs and runs in only one step.

Did you test this two-step solution? What on earth were you expecting it to do, if it got through basic validation of control cards?
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   This topic is locked: you cannot edit posts or make replies.    IBMMAINFRAMES.com Support Forums -> JCL & VSAM All times are GMT + 6 Hours
Goto page Previous  1, 2
Page 2 of 2

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Removing Hyphens and Dots using Syncsort PORYES SYNCSORT 8 Tue Apr 28, 2020 9:52 pm
No new posts Sort on different fields, removing du... MOHIT KUMAR GUPTA SYNCSORT 8 Thu Apr 16, 2020 5:54 pm
No new posts On a team working on a new project fo... zCMHII All Other Mainframe Topics 2 Mon Apr 06, 2020 9:15 pm
No new posts ICETOOL Sorting and Discarding DUPS Harold Barnes SYNCSORT 11 Sat Dec 14, 2019 5:55 am
This topic is locked: you cannot edit posts or make replies. Removing duplicate record based on th... Nilanjan Sikdar DFSORT/ICETOOL 4 Mon Jul 22, 2019 8:29 pm

Facebook
Back to Top
 
Job Vacancies | Forum Rules | Bookmarks | Subscriptions | FAQ | Polls | Contact Us