You are specifying both EQUALS and SUM FIELDS=NONE. Drop at least one of them, preferrably both.
Check in your Syncsort manual, I'm sure they have information on sorting large files. I would try to get the whole key together, and such that it would be possible to sort it all ASCENDING (what is producing the 900,000,000+ records per day, do you have any influence over that?).
Is it simply reducing CPU for charging, or do you want better thoughput?
Joined: 06 Jun 2008 Posts: 8411 Location: Dubuque, Iowa, USA
is there any way to reduce CPU time so that it should reduce the job's run cost.
Be aware that "run cost" is not standard across sites. Some sites use chargeback schemes to recover the mainframe cost while other sites do not. If a site does not use a chargeback scheme, then any given job will use resources (CPU time, I/O, elapsed time) but the "cost" of the job in accounting terms would be ZERO and that could not be reduced.
Furhtermore, chargeback schemes can be simple or very complex -- again, depending upon the site. If your site's chargeback scheme places a relatively high value on CPU time, then you may not be able to do much to reduce the "cost" since it's going to take a certain amount of CPU time to process that many records -- and the sort products tend to be pretty optimized already, so there's not much short of completely redesigning your processing that will cut the CPU usage.
As Bill said, EQUALS and SUM FIELDS have opposite meaning.
1. loose EQUALS first.
Syncsort 1.3 Programmers Guide wrote:
The EQUALS PARM instructs the sort/merge to preserve the order of equal-keyed records.
EQUALS will have a slight but generally significant impact on sort performance. By making
EQUALS available on an individual sort basis, SyncSort makes this programming
option available where it is needed, without imposing it on the installation’s more routine
jobs. For sort efficiency, use EQUALS only where the preservation of the input order of
equal-keyed records is important.
2. check if you really have duplicated records. If no loose the SUM card.
3. Specifying large enough SORTWK areas on different volumes may help too.
4. You really should open the Guide and check the available options...
Syncsort 1.3 Programmers Guide wrote:
A sort is considered to finely tuned when WER124I reports an overallocation factor between 1.00 and 1.50.
EQUALS will have a slight but generally significant impact on sort performance
I like that. If the weather forecast was for "slight but generally significant falls of snow" you'd not know whether to take an umbrella or a shovel to work with you.
Taking the EQUALS off, I'm sure you'll notice the difference.
If you really have duplicates, and you really have to have the physically latest record as the representative for that key, then there is a bit of a quandry. If there is something on the record which indicates the latest one, then post-process in whichever program is reading the file afterwards. If you have nothing, look at the viability of getting something, compared to the CPU-cost saving of remval.
You specify two seperate fields in the sort card which are contiguous. Specify them as one field. You have a PD, dislocated from the rest of the key, which is sorted descending. Look at the possibility of the producer making it negative, or "subtracted" from 999 and try to get it moved adjacent to the rest of the key, so it can all be specified in one statement.
You can prepare the records like that in an extract from your file and do some comparisons, comparing CPU usage. If the recored can be made smaller, that will help, but you'll have to find out how much. If there are records that can be excluded because they are not needed (other than duplicates) then do it as early as possible.
What about any processing that follows the sort, all the way through? Anything dumb in programs using that file is going to be significantly multiplied by the sheer size of the file. You've probably got bigger savings, as a percentage, in the rest of the processing, the Sort is just a highlight because it is a heavy user of CPU.
I'm sure if you contact Syncsort they will be willing to assist you with tuning. If you search for Allissa's posts in this forum you'll find some e-mails you can enquire on.