|
View previous topic :: View next topic
|
| Author |
Message |
ribu_thms
New User
Joined: 29 Dec 2020
Posts: 6
Location: Germany
|
|
|
|
| I am currently working on an assignment to improve the performance of one Batch Job. This Job has lot of SORT statements which performs a simply SORT on the input data. The Job was initially using utility IGHRCO00 with multiple SORTWRK files. I replaced it with SORT utility and removed all the SORTWRK files. As expected, the Job runs without any problem but there was no impact on the Performance of the Job. Can you please suggest to me some alternative to improve the CPU Performance of these SORT steps? |
|
| Back to top |
|
 |
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1335
Location: Bamberg, Germany
|
|
|
|
| IGHRCO00 is just an ALIAS of ICEMAN. |
|
| Back to top |
|
|
Nic Clouston
Global Moderator
Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK
|
|
|
|
| Generally sort is pretty well optimised so you best option on cutting wall-clock time is to split the job into several jobs and run in parallel. If you are not seeking to cut elapsed time but reduce cpu and/or i/o then you would need to look at your sort control cards. |
|
| Back to top |
|
|
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1335
Location: Bamberg, Germany
|
|
|
|
| @ribu_thms: Provide us with the OPTIONS in use, may be that can shed some light on your issue (if it is one). Also see freshly introduced ZSORT if that's something you can utilize. |
|
| Back to top |
|
|
Rohit Umarjikar
Global Moderator
Joined: 21 Sep 2010
Posts: 3076
Location: NYC,USA
|
|
|
|
Welcome!
You would really have to go thru each step within the job to see what all steps are really taking more cpu or has elapsed time and then analyze the total volume and the sort card that you are processing.
Second you need to draw a line and decide what’s the deviation of cpu or elapsed time from the expectation and justify why so.
Once you share these details then we could help better otherwise it’s pretty generic question which has generic answers. |
|
| Back to top |
|
|
ribu_thms
New User
Joined: 29 Dec 2020
Posts: 6
Location: Germany
|
|
|
|
Many thanks to all for your valuable inputs so far.
So the input file has about 360 million data. The current PGM IGHRCO00 sorts this data in 11 mins with a CPU processing time of around 2 min 53 seconds. |
|
| Back to top |
|
|
Rohit Umarjikar
Global Moderator
Joined: 21 Sep 2010
Posts: 3076
Location: NYC,USA
|
|
|
|
| And that’s sounds too much, I think that’s usual? Do you really need this sorting ? Can that be avoided? |
|
| Back to top |
|
|
Nic Clouston
Global Moderator
Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK
|
|
|
|
| How many bytes/fields are you sorting on? Are any fields adjacent and sorted similarly? They couldbe sorted as one field. |
|
| Back to top |
|
|
enrico-sorichetti
Superior Member
Joined: 14 Mar 2007
Posts: 10888
Location: italy
|
|
|
|
| Quote: |
| Are any fields adjacent and sorted similarly? They could be sorted as one field. |
Once upon a time I had a chat with Frank about it and He told me that the impact would be negligible |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2141
Location: USA
|
|
|
|
Instead of “micro-tuning” of each SORT step I would recommend to review the whole process logic in general.
Often those JCLs with hundreds steps are being “improved” for many years by 5-50 different people. Each of them has no time, or no desire, to analyze the whole process, but instead adds 3-5-10 new steps in the middle of many others. Very often the same dataset is re-sorted many times in one job, back and forth, every time just to produce a single report or a new set of records. After careful review many steps can be either combined, or optimized, or the result of one SORT can be used for several processing steps requiring the same sort order, without re-sorting it again, and again.
Such careful review of general algorithm might give more serious performance improvement. Not always, but very often. |
|
| Back to top |
|
|
ribu_thms
New User
Joined: 29 Dec 2020
Posts: 6
Location: Germany
|
|
|
|
| enrico-sorichetti wrote: |
| Quote: |
| Are any fields adjacent and sorted similarly? They could be sorted as one field. |
Once upon a time I had a chat with Frank about it and He told me that the impact would be negligible |
Yes. I have already tested this. There was no impact on the Processing time. Infact, sorting adjacent fields as one field slight increased the CPU time.
In one example, the input file had the following SYSIN parameter
SORT FIELDS=(1,12,CH,A,14,2,CH,A)
the 13th field was always a constant character 'A'. So I changed the SORT fields as
SORT FIELDS=(1,15,CH,A)
Surprisingly, the CPU time was increased by about 10 seconds. |
|
| Back to top |
|
|
ribu_thms
New User
Joined: 29 Dec 2020
Posts: 6
Location: Germany
|
|
|
|
| sergeyken wrote: |
Instead of “micro-tuning” of each SORT step I would recommend to review the whole process logic in general.
Often those JCLs with hundreds steps are being “improved” for many years by 5-50 different people. Each of them has no time, or no desire, to analyze the whole process, but instead adds 3-5-10 new steps in the middle of many others. Very often the same dataset is re-sorted many times in one job, back and forth, every time just to produce a single report or a new set of records. After careful review many steps can be either combined, or optimized, or the result of one SORT can be used for several processing steps requiring the same sort order, without re-sorting it again, and again.
Such careful review of general algorithm might give more serious performance improvement. Not always, but very often. |
I have reviewed the entire Job and unfortunately the SORT statements cannot be avoided. |
|
| Back to top |
|
|
Rohit Umarjikar
Global Moderator
Joined: 21 Sep 2010
Posts: 3076
Location: NYC,USA
|
|
|
|
Some experiments-
1. Split the process into 4-5 jobs and Merge later. Elapsed time will reduce and I hope for CPU ( may be a slight reduction) as well.
2. You can try INREC BUILD( ) , restrict the record to max actual size instead of copying everything, see if that helps.
3. Try adding REGION on this step with higher value.
4. Contact IBM and share your problem.
Topic moved to DFSORT. |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2141
Location: USA
|
|
|
|
| ribu_thms wrote: |
| sergeyken wrote: |
Instead of “micro-tuning” of each SORT step I would recommend to review the whole process logic in general.
Often those JCLs with hundreds steps are being “improved” for many years by 5-50 different people. Each of them has no time, or no desire, to analyze the whole process, but instead adds 3-5-10 new steps in the middle of many others. Very often the same dataset is re-sorted many times in one job, back and forth, every time just to produce a single report or a new set of records. After careful review many steps can be either combined, or optimized, or the result of one SORT can be used for several processing steps requiring the same sort order, without re-sorting it again, and again.
Such careful review of general algorithm might give more serious performance improvement. Not always, but very often. |
I have reviewed the entire Job and unfortunately the SORT statements cannot be avoided. |
There cannot be any general, abstract solution for this issue. You need to carefully review the whole process specific exclusively to your case.
It makes no sense to ask for such “common and general” magic trick from this forum, while giving absolutely no specific details related to your situation. |
|
| Back to top |
|
|
ribu_thms
New User
Joined: 29 Dec 2020
Posts: 6
Location: Germany
|
|
|
|
| Rohit Umarjikar wrote: |
Some experiments-
1. Split the process into 4-5 jobs and Merge later. Elapsed time will reduce and I hope for CPU ( may be a slight reduction) as well.
2. You can try INREC BUILD( ) , restrict the record to max actual size instead of copying everything, see if that helps.
3. Try adding REGION on this step with higher value.
4. Contact IBM and share your problem.
Topic moved to DFSORT. |
Many thanks Rohit. I will try these options. Really appreciated. |
|
| Back to top |
|
|
ribu_thms
New User
Joined: 29 Dec 2020
Posts: 6
Location: Germany
|
|
|
|
| sergeyken wrote: |
| ribu_thms wrote: |
| sergeyken wrote: |
Instead of “micro-tuning” of each SORT step I would recommend to review the whole process logic in general.
Often those JCLs with hundreds steps are being “improved” for many years by 5-50 different people. Each of them has no time, or no desire, to analyze the whole process, but instead adds 3-5-10 new steps in the middle of many others. Very often the same dataset is re-sorted many times in one job, back and forth, every time just to produce a single report or a new set of records. After careful review many steps can be either combined, or optimized, or the result of one SORT can be used for several processing steps requiring the same sort order, without re-sorting it again, and again.
Such careful review of general algorithm might give more serious performance improvement. Not always, but very often. |
I have reviewed the entire Job and unfortunately the SORT statements cannot be avoided. |
There cannot be any general, abstract solution for this issue. You need to carefully review the whole process specific exclusively to your case.
It makes no sense to ask for such “common and general” magic trick from this forum, while giving absolutely no specific details related to your situation. |
I agree that my query is a generic one. In this forum, I have also tried to give some examples. I have received some good information from this forum which has helped me. I was never expecting a magic trick. But I understand your point and also agree with your point that multiple people over the years have done changes without much thought over the futuristic impact. Many thanks for your reply. |
|
| Back to top |
|
|
Rohit Umarjikar
Global Moderator
Joined: 21 Sep 2010
Posts: 3076
Location: NYC,USA
|
|
|
|
| ribu_thms wrote: |
| Rohit Umarjikar wrote: |
Some experiments-
1. Split the process into 4-5 jobs and Merge later. Elapsed time will reduce and I hope for CPU ( may be a slight reduction) as well.
2. You can try INREC BUILD( ) , restrict the record to max actual size instead of copying everything, see if that helps.
3. Try adding REGION on this step with higher value.
4. Contact IBM and share your problem.
Topic moved to DFSORT. |
Many thanks Rohit. I will try these options. Really appreciated. |
Welcome! Let us know if they do any improvements, it may help others in future. Thanks. |
|
| Back to top |
|
|
|
|
