IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

DFSORT is running longer due to huge Input volume
IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic Quick References
View previous topic :: View next topic  
Author Message
V S Amarendra Reddy

Active User


Joined: 13 Sep 2006
Posts: 216
Location: USA
PostPosted: Tue Dec 18, 2012 12:36 am
 Reply with quote
Hi,

My job has one input file and an output file. It was coded sometime ago and it is running too long. Can someone please clarify me, if it can be tuned further to reduce the cpu time and run time? The details are as below. These records are sorted and stored in DB (Outline of requirement).

Control card:
Code:

SORT FIELDS=(313,4,BI,A),DYNALLOC=(3390,255),
FILSZ=E500000000                             


DFSORT messages: As shown the input count is about 300-400 Million records.

Code:

ICE088I 1 F6435HMF.JHMF0001.STEP07  , INPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB
ICE093I 0 MAIN STORAGE = (MAX,171512712,103230918)       
ICE156I 0 MAIN STORAGE ABOVE 16MB = (101150720,101150720)
ICE090I 0 OUTPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB           
ICE055I 0 INSERT 0, DELETE 0                                       
ICE054I 0 RECORDS - IN: 372774789, OUT: 372774789                   
ICE134I 0 NUMBER OF BYTES SORTED: 201298386060                     
ICE253I 0 RECORDS SORTED - PROCESSED: 372774789, EXPECTED: 372774789
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 4804410 , TRACKS USED: 3506010
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 33553680K BYTES   
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES           


The step run statistics : 18.55 Min of CPU time and about 2 hours of execution time is taken to complete this job.

Code:
PROCSTEP    RC   EXCP    CPU    SRB  CLOCK
   STEP07      00  4929K    18.55   5.94 110.42


Can someone please help me out if the time taken can be reduced?

Please let me know if you need additional information.

Regards
Amar
Back to top
View user's profile Send private message
Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1788
Location: Bloomington, IL
PostPosted: Tue Dec 18, 2012 12:51 am
 Reply with quote
Remember that CPU time and elapsed/wall time have but a tenuous relationship to each other. You'll have to have your performance workgroup look at the overall picture.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
PostPosted: Tue Dec 18, 2012 12:57 am
 Reply with quote
Hello,

Quote:
It was coded sometime ago and it is running too long.
By what definition? If you have to sort 300-400million records, it Will take considerable time.

How is the input created? Is this mostly records from a previous run to be combined with some new records in the needed order?

If so, consider sorting the new records and using MERGE to combine the data.

What media is used for the input/output?
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix
PostPosted: Tue Dec 18, 2012 3:57 am
 Reply with quote
I suggest you follow this advice. Make sure you send the FULL sysout, not just what you showed us.
Back to top
View user's profile Send private message
V S Amarendra Reddy

Active User


Joined: 13 Sep 2006
Posts: 216
Location: USA
PostPosted: Tue Dec 18, 2012 4:03 pm
 Reply with quote
This is total SYSOUT. The input/output are sequential files.

Code:

1ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
 ICE751I 0 C5-K76982 C6-K90026 C7-K58148 C8-K67572 E4-K58148 C9-BASE   E5-K76585 E6-K58148 C4-K58148 E7-K70685
 ICE143I 0 BLOCKSET     SORT  TECHNIQUE SELECTED
 ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
 ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 12:11 ON FRI DEC 14, 2012 -
0           SORT FIELDS=(313,4,BI,A),DYNALLOC=(3390,255),
            FILSZ=E500000000
 ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
 ICE751I 0 C5-K76982 C6-K90026 C7-K58148 C8-K67572 E4-K58148 C9-BASE   E5-K76585 E6-K58148 C4-K58148 E7-K70685
 ICE193I 0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
 ICE252I 1 PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS
 ICE088I 1 F6435HMF.JHMF0001.STEP07  , INPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB
 ICE093I 0 MAIN STORAGE = (MAX,171512712,103230918)
 ICE156I 0 MAIN STORAGE ABOVE 16MB = (101150720,101150720)
 ICE127I 0 OPTIONS: OVFLO=RC4 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
 ICE128I 0 OPTIONS: SIZE=171512712,MAXLIM=2097512,MINLIM=512000,EQUALS=Y,LIST=Y,ERET=ABEND,MSGDDN=SYSOUT
 ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=SHORT,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(3390    ,255),ABCODE=MSG
 ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
 ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DSA=512
 ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
 ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
 ICE235I 0 OPTIONS: NULLOUT=RC0
 ICE236I 0 OPTIONS: DYNAPCT=OLD,MOWRK=Y
 ICE084I 0 BSAM ACCESS METHOD USED FOR SORTOUT
 ICE084I 0 BSAM ACCESS METHOD USED FOR SORTIN
 ICE750I 0 DC 201298386060 TC 0 CS DSVVV KSZ 8 VSZ 8
 ICE752I 0 FSZ=372774789 RC  IGN=500000000 E  AVG=544 0  WSP=263388687 C  DYN=9602978 24544
 ICE751I 1 DE-K61787 D5-K58148 D3-K58148 D7-K58148 E8-K70685
 ICE090I 0 OUTPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB
 ICE055I 0 INSERT 0, DELETE 0
 ICE054I 0 RECORDS - IN: 372774789, OUT: 372774789
 ICE134I 0 NUMBER OF BYTES SORTED: 201298386060
 ICE253I 0 RECORDS SORTED - PROCESSED: 372774789, EXPECTED: 372774789
 ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 4804410 , TRACKS USED: 3506010
 ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
 ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES
 ICE180I 0 HIPERSPACE STORAGE USED = 33553680K BYTES
 ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
 ICE052I 0 END OF DFSORT
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix
PostPosted: Tue Dec 18, 2012 5:16 pm
 Reply with quote
Did you need EQUALS? Can your explain the processing?
Back to top
View user's profile Send private message
sqlcode1

Active Member


Joined: 08 Apr 2010
Posts: 577
Location: USA
PostPosted: Tue Dec 18, 2012 6:20 pm
 Reply with quote
V S Amarendra Reddy,
Code:
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 4804410 , TRACKS USED: 3506010

Not sure if you need max. number in DYNALLOC. Have you tried with smaller number?

Thanks,
Back to top
View user's profile Send private message
V S Amarendra Reddy

Active User


Joined: 13 Sep 2006
Posts: 216
Location: USA
PostPosted: Tue Dec 18, 2012 9:28 pm
 Reply with quote
Hi Bill,

EQUALS is not really required. The process is they are files unloaded from a table and these will be input to a program which will do some process and produces final records which should be loaded into another table.


Sqlcode1,

Since the records that are coming in are dynamic, they have kept the DYNALLOC parm as max. Will there be improvement if we reduce the parm?


Regards
Amar
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix
PostPosted: Tue Dec 18, 2012 9:45 pm
 Reply with quote
OK, so try with NOEQUALS on the OPTION or on the SORT, which should reduce CPU usage.

If EQUALS is your site default, there may well be other Sort steps which would benefit - any Sort which does not need to preserve the input order when keys are equal can benefit from NOEQUALS as far as CPU usage is concerned.

Your input and output are tapes? Ensure that those are handled effectively, else elapsed time will be increased for any "mount" waits. If DASD, you are loosing a lot through not having optimum blocksizes, but I'm assuming it is tape.
Back to top
View user's profile Send private message
V S Amarendra Reddy

Active User


Joined: 13 Sep 2006
Posts: 216
Location: USA
PostPosted: Tue Dec 18, 2012 11:21 pm
 Reply with quote
Thank you Bill. I will try with No EQUALS option. The files are on DASD. In JCL, we are keeping BLKSIZE as 0 in DCB parameters, so that SMS will choose the optimum block size before the file is being allocated.


Code:

//SORTOUT  DD DSN=HLQ.output,     
//        DISP=(,CATLG),UNIT=(,32),SPACE=(CYL,(750,750),RLSE)
//        DCB=(RECFM=FB,LRECL=540,BLKSIZE=0),DATACLAS=DCCOMP
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix
PostPosted: Wed Dec 19, 2012 12:50 am
 Reply with quote
Code:
ICE088I 1 F6435HMF.JHMF0001.STEP07  , INPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB
 [...]
ICE090I 0 OUTPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB


I suggest you check your datasets.
Back to top
View user's profile Send private message
Pandora-Box

Global Moderator


Joined: 07 Sep 2006
Posts: 1592
Location: Andromeda Galaxy
PostPosted: Wed Dec 19, 2012 1:09 am
 Reply with quote
Also do you have the option of running jobs in parallel?
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose
PostPosted: Wed Dec 19, 2012 1:19 am
 Reply with quote
V S Amarendra Reddy wrote:
This is total SYSOUT. The input/output are sequential files.

V S Amarendra Reddy,

If you need help, Please add //SORTDIAG DD DUMMY to your JCL and re-run the job and send us the complete sysout to DFSORT hot line: dfsort@us.ibm.com and our performance expert will look into it.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 

Similar Topics
Topic Forum Replies
No new posts TRIM everything from input, output co... DFSORT/ICETOOL 1
No new posts Running REXX through JOB CLIST & REXX 13
No new posts Extracting Variable decimal numbers f... DFSORT/ICETOOL 17
No new posts Modifying Date Format Using DFSORT DFSORT/ICETOOL 9
No new posts Replace Multiple Field values to Othe... DFSORT/ICETOOL 12
Search our Forums:

Back to Top
IBMMainframes.com is not an official and/or affiliated with IBM® in anyway
Board Rules | FAQ | Downloads | Wiki | SiteMap | Contact Us