I have a flat file which contains huge records(500-600krecords).I have to copy this into a KSDS .The flat file is being deleted and recreated everyday with fresh data . this is being appended into KSDS everyday(also needs to be updated in case of duplicate key). So during the month end the KSDS will be having the entire months records.
But i just cant use SORT FIELDS=COPY, since the keys might be duplicate.
Can anyone suggest how to include some condition so as to check for existence of key and to write/rewrite the KSDS.
It's not clear from your description whether the first record of duplicates is the latest record or the last record of duplicates is the latest record.
If you want to keep only the first record of duplicates, you can use ICETOOL's SELECT operator with the FIRST operand. If you want to keep only the last record of duplicates you can use SELECT with LAST.