Welcome to the forums. I guess you want to do this using the SORT utility installed in your shop - DFSORT/SyncSort etc . Please be aware that SYNCSORT related questions are discussed in the JCL forum and the DFSORT related queries in the DFSORT part of this forum. If you mention which product you are having, it'll help the moderators to move this topic to the appropriate forum.
Joined: 23 Nov 2006 Posts: 19270 Location: Inside the Matrix
We need this in one step as we cannot afford multiple sort steps with huge files.
The number of steps is not the performance issue - the number of passes of the data is the issue. There are many posted solutions using a single step, but multiple passes of the data (which is what you actually need to avoid if possible).
What percent of the records are discarded as duplicatess in the first pass you posted? It may be that the second pass of the "dups removed" data is not so large. . .