IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Which SORT utility can improve the Performance of the JOB


IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
ribu_thms

New User


Joined: 29 Dec 2020
Posts: 6
Location: Germany

PostPosted: Tue Dec 29, 2020 3:01 pm
Reply with quote

I am currently working on an assignment to improve the performance of one Batch Job. This Job has lot of SORT statements which performs a simply SORT on the input data. The Job was initially using utility IGHRCO00 with multiple SORTWRK files. I replaced it with SORT utility and removed all the SORTWRK files. As expected, the Job runs without any problem but there was no impact on the Performance of the Job. Can you please suggest to me some alternative to improve the CPU Performance of these SORT steps?
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1308
Location: Bamberg, Germany

PostPosted: Tue Dec 29, 2020 3:20 pm
Reply with quote

IGHRCO00 is just an ALIAS of ICEMAN.
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK

PostPosted: Tue Dec 29, 2020 4:22 pm
Reply with quote

Generally sort is pretty well optimised so you best option on cutting wall-clock time is to split the job into several jobs and run in parallel. If you are not seeking to cut elapsed time but reduce cpu and/or i/o then you would need to look at your sort control cards.
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1308
Location: Bamberg, Germany

PostPosted: Wed Dec 30, 2020 11:40 am
Reply with quote

@ribu_thms: Provide us with the OPTIONS in use, may be that can shed some light on your issue (if it is one). Also see freshly introduced ZSORT if that's something you can utilize.
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3075
Location: NYC,USA

PostPosted: Wed Dec 30, 2020 1:03 pm
Reply with quote

Welcome!
You would really have to go thru each step within the job to see what all steps are really taking more cpu or has elapsed time and then analyze the total volume and the sort card that you are processing.
Second you need to draw a line and decide what’s the deviation of cpu or elapsed time from the expectation and justify why so.

Once you share these details then we could help better otherwise it’s pretty generic question which has generic answers.
Back to top
View user's profile Send private message
ribu_thms

New User


Joined: 29 Dec 2020
Posts: 6
Location: Germany

PostPosted: Wed Dec 30, 2020 3:18 pm
Reply with quote

Many thanks to all for your valuable inputs so far.

So the input file has about 360 million data. The current PGM IGHRCO00 sorts this data in 11 mins with a CPU processing time of around 2 min 53 seconds.
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3075
Location: NYC,USA

PostPosted: Wed Dec 30, 2020 7:05 pm
Reply with quote

And that’s sounds too much, I think that’s usual? Do you really need this sorting ? Can that be avoided?
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK

PostPosted: Wed Dec 30, 2020 7:39 pm
Reply with quote

How many bytes/fields are you sorting on? Are any fields adjacent and sorted similarly? They couldbe sorted as one field.
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10879
Location: italy

PostPosted: Wed Dec 30, 2020 8:08 pm
Reply with quote

Quote:
Are any fields adjacent and sorted similarly? They could be sorted as one field.

Once upon a time I had a chat with Frank about it and He told me that the impact would be negligible
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2120
Location: USA

PostPosted: Wed Dec 30, 2020 8:48 pm
Reply with quote

Instead of “micro-tuning” of each SORT step I would recommend to review the whole process logic in general.

Often those JCLs with hundreds steps are being “improved” for many years by 5-50 different people. Each of them has no time, or no desire, to analyze the whole process, but instead adds 3-5-10 new steps in the middle of many others. Very often the same dataset is re-sorted many times in one job, back and forth, every time just to produce a single report or a new set of records. After careful review many steps can be either combined, or optimized, or the result of one SORT can be used for several processing steps requiring the same sort order, without re-sorting it again, and again.

Such careful review of general algorithm might give more serious performance improvement. Not always, but very often.
Back to top
View user's profile Send private message
ribu_thms

New User


Joined: 29 Dec 2020
Posts: 6
Location: Germany

PostPosted: Thu Dec 31, 2020 8:04 pm
Reply with quote

enrico-sorichetti wrote:
Quote:
Are any fields adjacent and sorted similarly? They could be sorted as one field.

Once upon a time I had a chat with Frank about it and He told me that the impact would be negligible


Yes. I have already tested this. There was no impact on the Processing time. Infact, sorting adjacent fields as one field slight increased the CPU time.

In one example, the input file had the following SYSIN parameter

SORT FIELDS=(1,12,CH,A,14,2,CH,A)

the 13th field was always a constant character 'A'. So I changed the SORT fields as

SORT FIELDS=(1,15,CH,A)

Surprisingly, the CPU time was increased by about 10 seconds.
Back to top
View user's profile Send private message
ribu_thms

New User


Joined: 29 Dec 2020
Posts: 6
Location: Germany

PostPosted: Thu Dec 31, 2020 8:07 pm
Reply with quote

sergeyken wrote:
Instead of “micro-tuning” of each SORT step I would recommend to review the whole process logic in general.

Often those JCLs with hundreds steps are being “improved” for many years by 5-50 different people. Each of them has no time, or no desire, to analyze the whole process, but instead adds 3-5-10 new steps in the middle of many others. Very often the same dataset is re-sorted many times in one job, back and forth, every time just to produce a single report or a new set of records. After careful review many steps can be either combined, or optimized, or the result of one SORT can be used for several processing steps requiring the same sort order, without re-sorting it again, and again.

Such careful review of general algorithm might give more serious performance improvement. Not always, but very often.



I have reviewed the entire Job and unfortunately the SORT statements cannot be avoided.
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3075
Location: NYC,USA

PostPosted: Thu Dec 31, 2020 9:14 pm
Reply with quote

Some experiments-
1. Split the process into 4-5 jobs and Merge later. Elapsed time will reduce and I hope for CPU ( may be a slight reduction) as well.
2. You can try INREC BUILD( ) , restrict the record to max actual size instead of copying everything, see if that helps.
3. Try adding REGION on this step with higher value.
4. Contact IBM and share your problem.

Topic moved to DFSORT.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2120
Location: USA

PostPosted: Thu Dec 31, 2020 9:31 pm
Reply with quote

ribu_thms wrote:
sergeyken wrote:
Instead of “micro-tuning” of each SORT step I would recommend to review the whole process logic in general.

Often those JCLs with hundreds steps are being “improved” for many years by 5-50 different people. Each of them has no time, or no desire, to analyze the whole process, but instead adds 3-5-10 new steps in the middle of many others. Very often the same dataset is re-sorted many times in one job, back and forth, every time just to produce a single report or a new set of records. After careful review many steps can be either combined, or optimized, or the result of one SORT can be used for several processing steps requiring the same sort order, without re-sorting it again, and again.

Such careful review of general algorithm might give more serious performance improvement. Not always, but very often.


I have reviewed the entire Job and unfortunately the SORT statements cannot be avoided.

There cannot be any general, abstract solution for this issue. You need to carefully review the whole process specific exclusively to your case.

It makes no sense to ask for such “common and general” magic trick from this forum, while giving absolutely no specific details related to your situation.
Back to top
View user's profile Send private message
ribu_thms

New User


Joined: 29 Dec 2020
Posts: 6
Location: Germany

PostPosted: Thu Dec 31, 2020 9:46 pm
Reply with quote

Rohit Umarjikar wrote:
Some experiments-
1. Split the process into 4-5 jobs and Merge later. Elapsed time will reduce and I hope for CPU ( may be a slight reduction) as well.
2. You can try INREC BUILD( ) , restrict the record to max actual size instead of copying everything, see if that helps.
3. Try adding REGION on this step with higher value.
4. Contact IBM and share your problem.

Topic moved to DFSORT.


Many thanks Rohit. I will try these options. Really appreciated.
Back to top
View user's profile Send private message
ribu_thms

New User


Joined: 29 Dec 2020
Posts: 6
Location: Germany

PostPosted: Thu Dec 31, 2020 10:02 pm
Reply with quote

sergeyken wrote:
ribu_thms wrote:
sergeyken wrote:
Instead of “micro-tuning” of each SORT step I would recommend to review the whole process logic in general.

Often those JCLs with hundreds steps are being “improved” for many years by 5-50 different people. Each of them has no time, or no desire, to analyze the whole process, but instead adds 3-5-10 new steps in the middle of many others. Very often the same dataset is re-sorted many times in one job, back and forth, every time just to produce a single report or a new set of records. After careful review many steps can be either combined, or optimized, or the result of one SORT can be used for several processing steps requiring the same sort order, without re-sorting it again, and again.

Such careful review of general algorithm might give more serious performance improvement. Not always, but very often.


I have reviewed the entire Job and unfortunately the SORT statements cannot be avoided.

There cannot be any general, abstract solution for this issue. You need to carefully review the whole process specific exclusively to your case.

It makes no sense to ask for such “common and general” magic trick from this forum, while giving absolutely no specific details related to your situation.


I agree that my query is a generic one. In this forum, I have also tried to give some examples. I have received some good information from this forum which has helped me. I was never expecting a magic trick. But I understand your point and also agree with your point that multiple people over the years have done changes without much thought over the futuristic impact. Many thanks for your reply.
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3075
Location: NYC,USA

PostPosted: Fri Jan 01, 2021 1:06 am
Reply with quote

ribu_thms wrote:
Rohit Umarjikar wrote:
Some experiments-
1. Split the process into 4-5 jobs and Merge later. Elapsed time will reduce and I hope for CPU ( may be a slight reduction) as well.
2. You can try INREC BUILD( ) , restrict the record to max actual size instead of copying everything, see if that helps.
3. Try adding REGION on this step with higher value.
4. Contact IBM and share your problem.

Topic moved to DFSORT.


Many thanks Rohit. I will try these options. Really appreciated.

Welcome! Let us know if they do any improvements, it may help others in future. Thanks.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 


Similar Topics
Topic Forum Replies
No new posts SORT CARD meaning please DFSORT/ICETOOL 3
No new posts Add condition to a FINDREP SORT card DFSORT/ICETOOL 4
No new posts Sort based on the record type DFSORT/ICETOOL 1
No new posts SORT JCL to merge multiple tow into s... DFSORT/ICETOOL 6
No new posts Use Copy Utility DB2 7
Search our Forums:

Back to Top