I have a requirement to remove duplicate record based on threshold limit. The limit will be parameterized and can be mentioned in control card. If the number of duplicate is more than the limit then the job should fail. Is it possible to do using DFSORT? Please help
I would think of this as one step solution.
1. Add duplicate counts per key at the end of the record using INREC
2. Using OUTFIL only include records whos counts form the INREC is greater than threshold limit (use JP1) and using NULLOFL set RC.
Joined: 10 May 2007 Posts: 2408 Location: Hampshire, UK
I think that should be BELOW the limit not GREATER than the limit.
Also, I am not clear on 2 points:
1 - are you removing ALL duplicates or just duplicates over the limit
2 - is the limit referring to the total duplicates in the data set or the number of duplicates per record.