IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Very Large Sort failed - Seeking recommendations


IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 6
Location: United States

PostPosted: Thu Jan 21, 2021 5:26 am
Reply with quote

Hello all y'all,

I was just assigned this problem. We have a job that requires sorting a very large file. The input file contains a little over 350 million 822 byte records. We are using DFSORT V2R4 in a z/OS 02.04.00 environment.

Our JCL is set up with SORTIN coming from TAPE3480, and SORTOUT going to TAPE3480. We have SORTWK01 - SORTWK92 allocated to SORTDA, SPACE=(CYL,(500,500),RLSE). Last week, production Abended with Sort Capacity Exceeded at 99% completion level. We knew it would eventually happen. Production Control added SORTWK93 - SORTWK99 to get through the problem. The successful sort took just over 6 hours wall clock and 9 minutes CPU time to complete.

Also, the sort card looks like this: SORT FIELDS=(1,8,A,9,8,A),FORMAT=BI.

Some of my questions are:

1- Does have 2 fields together as separate Sort Fields make it less efficient? Should I change it to (1,16,A),FORMAT=BI? Is the BI format helping?

2- I know I can increase the number of Sort Works to 255, using names like SORTWKA1 - SORTWKZ9. How many should I add?

3-If I add more Sort Works, should I reduce the primary allocation?

4- Should I split the file into smaller sub pieces, such as 10% of the input file, do a sort on each file, then merge them all together?

5- Should I reduce the Sort Works to 32 and have them go to TAPE3480?

6- Other suggestions?

Thanks in advance for sharing your wisdom!
Back to top
View user's profile Send private message
dneufarth

Active User


Joined: 27 Apr 2005
Posts: 336
Location: Inside the SPEW (Cincinnati OH USA)

PostPosted: Thu Jan 21, 2021 6:41 am
Reply with quote

Been awhile ago for me, but this may help

www.ibm.com/search?lang=en&cc=us&q=Sorting%20huge%20files%20dfsort
Back to top
View user's profile Send private message
Joerg.Findeisen

Active Member


Joined: 15 Aug 2015
Posts: 623
Location: Bamberg, Germany

PostPosted: Thu Jan 21, 2021 11:18 am
Reply with quote

Remove static SORTWK from JCL, use DYNALLOC=(SYSALLDA,<n>) instead. Provide extra separate storage for the SORTWK datasets as they are some kind of special. Make use of DYNAPCT=<n> and check if you are capable to use the freshly introduced ZSORT option.
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 2790
Location: NYC,USA

PostPosted: Thu Jan 21, 2021 5:57 pm
Reply with quote

Option 4 is a long term solution to have it . Year on year volume may get increase and you keep getting these abends.

Also, try providing higher REGION on this step, talk to your storage team to see if they have STORCLAS options to use instead of these SORTWK.

Reading this might help further-
www.ibm.com/support/pages/system/files/inline-files/$FILE/SORTASKP.pdf
Back to top
View user's profile Send private message
Joerg.Findeisen

Active Member


Joined: 15 Aug 2015
Posts: 623
Location: Bamberg, Germany

PostPosted: Thu Jan 21, 2021 6:15 pm
Reply with quote

STORCLAS won't do anything. Try as I say - separate SGRP for those DSN.
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 2790
Location: NYC,USA

PostPosted: Thu Jan 21, 2021 7:35 pm
Reply with quote

I still prefer option-4 solution, full proof and long term way.
we use STORCLAS=COMPRESS when we had these type of issues but I don't recall what else was changed along with this. I don't disagree to try with DYNALLOC=(SYSALLDA,<n>).
Code:
OPTION DYNALLOC=(,350),FILSZ=E350000000

Add this to your JCL -
Code:
//SORTDIAG DD DUMMY

If nothing works then send these details to IBM to suggest something which will work else option 4.
Back to top
View user's profile Send private message
Joerg.Findeisen

Active Member


Joined: 15 Aug 2015
Posts: 623
Location: Bamberg, Germany

PostPosted: Fri Jan 22, 2021 12:03 pm
Reply with quote

Skip the FILSZ parm, DFSORT knows much more about the data by it's own mechanism. The //SORTDIAG should have SYSOUT=* to provide data for analysis.
Back to top
View user's profile Send private message
Pete Wilson

Active Member


Joined: 31 Dec 2009
Posts: 509
Location: London

PostPosted: Fri Jan 22, 2021 10:43 pm
Reply with quote

If you must use SORTWK's (or any systemp file) they cannot be compressed, or extended format. They work better as Large Format files which can grow to any size on the volumes within their max 16 extents.
(DSNTYPE=LARGE in the JCL)

I have found the below works almost every time for very large sorts (e.g. >9 billion records in one case) and without any SORTWK DD's specified. DYNALLOC and REGION can be adjusted according to available resources.

//*
//* FOR REALLY LARGE FILES THE BELOW EXEC PARMS ARE OPTIMAL
//*
//* EXEC PGM=SORT,PARM='DYNALLOC=(,24),DSPSIZE=MAX',REGION=512M
//*
//* NOTES:
//*
//* 1. THE DYNALLOC VALUE SHOULD NOT EXCEED HALF THE NUMBER OF
//* WORKPOOL VOLUMES.
//*
//* 2. REGION SHOULD HAVE A LARGE VALUE TO ALLOW MORE IN-STORAGE
//* CONTROL INFORMATION USED BY THE SORT.
//*
//* 3. DSPMAX ALLOCATES THE LARGEST POSSIBLE DATA-SPACE
//*
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 


Similar Topics
Topic Forum Replies
This topic is locked: you cannot edit posts or make replies. Sort to include records of file 2 int... Java & MQSeries 1
No new posts Converting unpacked fields to pack us... SYNCSORT 4
No new posts SORT for dynamic trailer record + CSV... DFSORT/ICETOOL 14
No new posts SORT HELP - SORT A COLUMN and GROUP B... DFSORT/ICETOOL 9
No new posts Sort w/OUTREC Question DFSORT/ICETOOL 2
Search our Forums:

Back to Top