Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

DFSORT is running longer due to huge Input volume

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL
View previous topic :: :: View next topic  
Author Message
V S Amarendra Reddy

Active User


Joined: 13 Sep 2006
Posts: 189
Location: INDIA

PostPosted: Tue Dec 18, 2012 12:36 am    Post subject: DFSORT is running longer due to huge Input volume
Reply with quote

Hi,

My job has one input file and an output file. It was coded sometime ago and it is running too long. Can someone please clarify me, if it can be tuned further to reduce the cpu time and run time? The details are as below. These records are sorted and stored in DB (Outline of requirement).

Control card:
Code:

SORT FIELDS=(313,4,BI,A),DYNALLOC=(3390,255),
FILSZ=E500000000                             


DFSORT messages: As shown the input count is about 300-400 Million records.

Code:

ICE088I 1 F6435HMF.JHMF0001.STEP07  , INPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB
ICE093I 0 MAIN STORAGE = (MAX,171512712,103230918)       
ICE156I 0 MAIN STORAGE ABOVE 16MB = (101150720,101150720)
ICE090I 0 OUTPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB           
ICE055I 0 INSERT 0, DELETE 0                                       
ICE054I 0 RECORDS - IN: 372774789, OUT: 372774789                   
ICE134I 0 NUMBER OF BYTES SORTED: 201298386060                     
ICE253I 0 RECORDS SORTED - PROCESSED: 372774789, EXPECTED: 372774789
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 4804410 , TRACKS USED: 3506010
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 33553680K BYTES   
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES           


The step run statistics : 18.55 Min of CPU time and about 2 hours of execution time is taken to complete this job.

Code:
PROCSTEP    RC   EXCP    CPU    SRB  CLOCK
   STEP07      00  4929K    18.55   5.94 110.42


Can someone please help me out if the time taken can be reduced?

Please let me know if you need additional information.

Regards
Amar
Back to top
View user's profile Send private message

Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1738
Location: Bloomington, IL

PostPosted: Tue Dec 18, 2012 12:51 am    Post subject:
Reply with quote

Remember that CPU time and elapsed/wall time have but a tenuous relationship to each other. You'll have to have your performance workgroup look at the overall picture.
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Tue Dec 18, 2012 12:57 am    Post subject:
Reply with quote

Hello,

Quote:
It was coded sometime ago and it is running too long.
By what definition? If you have to sort 300-400million records, it Will take considerable time.

How is the input created? Is this mostly records from a previous run to be combined with some new records in the needed order?

If so, consider sorting the new records and using MERGE to combine the data.

What media is used for the input/output?
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Tue Dec 18, 2012 3:57 am    Post subject: Reply to: DFSORT is running longer due to huge Input volume
Reply with quote

I suggest you follow this advice. Make sure you send the FULL sysout, not just what you showed us.
Back to top
View user's profile Send private message
V S Amarendra Reddy

Active User


Joined: 13 Sep 2006
Posts: 189
Location: INDIA

PostPosted: Tue Dec 18, 2012 4:03 pm    Post subject: Reply to: DFSORT is running longer due to huge Input volume
Reply with quote

This is total SYSOUT. The input/output are sequential files.

Code:

1ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
 ICE751I 0 C5-K76982 C6-K90026 C7-K58148 C8-K67572 E4-K58148 C9-BASE   E5-K76585 E6-K58148 C4-K58148 E7-K70685
 ICE143I 0 BLOCKSET     SORT  TECHNIQUE SELECTED
 ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
 ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 12:11 ON FRI DEC 14, 2012 -
0           SORT FIELDS=(313,4,BI,A),DYNALLOC=(3390,255),
            FILSZ=E500000000
 ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
 ICE751I 0 C5-K76982 C6-K90026 C7-K58148 C8-K67572 E4-K58148 C9-BASE   E5-K76585 E6-K58148 C4-K58148 E7-K70685
 ICE193I 0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
 ICE252I 1 PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS
 ICE088I 1 F6435HMF.JHMF0001.STEP07  , INPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB
 ICE093I 0 MAIN STORAGE = (MAX,171512712,103230918)
 ICE156I 0 MAIN STORAGE ABOVE 16MB = (101150720,101150720)
 ICE127I 0 OPTIONS: OVFLO=RC4 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
 ICE128I 0 OPTIONS: SIZE=171512712,MAXLIM=2097512,MINLIM=512000,EQUALS=Y,LIST=Y,ERET=ABEND,MSGDDN=SYSOUT
 ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=SHORT,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(3390    ,255),ABCODE=MSG
 ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
 ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DSA=512
 ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
 ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
 ICE235I 0 OPTIONS: NULLOUT=RC0
 ICE236I 0 OPTIONS: DYNAPCT=OLD,MOWRK=Y
 ICE084I 0 BSAM ACCESS METHOD USED FOR SORTOUT
 ICE084I 0 BSAM ACCESS METHOD USED FOR SORTIN
 ICE750I 0 DC 201298386060 TC 0 CS DSVVV KSZ 8 VSZ 8
 ICE752I 0 FSZ=372774789 RC  IGN=500000000 E  AVG=544 0  WSP=263388687 C  DYN=9602978 24544
 ICE751I 1 DE-K61787 D5-K58148 D3-K58148 D7-K58148 E8-K70685
 ICE090I 0 OUTPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB
 ICE055I 0 INSERT 0, DELETE 0
 ICE054I 0 RECORDS - IN: 372774789, OUT: 372774789
 ICE134I 0 NUMBER OF BYTES SORTED: 201298386060
 ICE253I 0 RECORDS SORTED - PROCESSED: 372774789, EXPECTED: 372774789
 ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 4804410 , TRACKS USED: 3506010
 ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
 ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES
 ICE180I 0 HIPERSPACE STORAGE USED = 33553680K BYTES
 ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
 ICE052I 0 END OF DFSORT
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Tue Dec 18, 2012 5:16 pm    Post subject: Reply to: DFSORT is running longer due to huge Input volume
Reply with quote

Did you need EQUALS? Can your explain the processing?
Back to top
View user's profile Send private message
sqlcode1

Active Member


Joined: 08 Apr 2010
Posts: 578
Location: USA

PostPosted: Tue Dec 18, 2012 6:20 pm    Post subject:
Reply with quote

V S Amarendra Reddy,
Code:
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 4804410 , TRACKS USED: 3506010

Not sure if you need max. number in DYNALLOC. Have you tried with smaller number?

Thanks,
Back to top
View user's profile Send private message
V S Amarendra Reddy

Active User


Joined: 13 Sep 2006
Posts: 189
Location: INDIA

PostPosted: Tue Dec 18, 2012 9:28 pm    Post subject:
Reply with quote

Hi Bill,

EQUALS is not really required. The process is they are files unloaded from a table and these will be input to a program which will do some process and produces final records which should be loaded into another table.


Sqlcode1,

Since the records that are coming in are dynamic, they have kept the DYNALLOC parm as max. Will there be improvement if we reduce the parm?


Regards
Amar
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Tue Dec 18, 2012 9:45 pm    Post subject: Reply to: DFSORT is running longer due to huge Input volume
Reply with quote

OK, so try with NOEQUALS on the OPTION or on the SORT, which should reduce CPU usage.

If EQUALS is your site default, there may well be other Sort steps which would benefit - any Sort which does not need to preserve the input order when keys are equal can benefit from NOEQUALS as far as CPU usage is concerned.

Your input and output are tapes? Ensure that those are handled effectively, else elapsed time will be increased for any "mount" waits. If DASD, you are loosing a lot through not having optimum blocksizes, but I'm assuming it is tape.
Back to top
View user's profile Send private message
V S Amarendra Reddy

Active User


Joined: 13 Sep 2006
Posts: 189
Location: INDIA

PostPosted: Tue Dec 18, 2012 11:21 pm    Post subject:
Reply with quote

Thank you Bill. I will try with No EQUALS option. The files are on DASD. In JCL, we are keeping BLKSIZE as 0 in DCB parameters, so that SMS will choose the optimum block size before the file is being allocated.


Code:

//SORTOUT  DD DSN=HLQ.output,     
//        DISP=(,CATLG),UNIT=(,32),SPACE=(CYL,(750,750),RLSE)
//        DCB=(RECFM=FB,LRECL=540,BLKSIZE=0),DATACLAS=DCCOMP
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Wed Dec 19, 2012 12:50 am    Post subject: Reply to: DFSORT is running longer due to huge Input volume
Reply with quote

Code:
ICE088I 1 F6435HMF.JHMF0001.STEP07  , INPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB
 [...]
ICE090I 0 OUTPUT LRECL = 540, BLKSIZE = 32400, TYPE = FB


I suggest you check your datasets.
Back to top
View user's profile Send private message
Pandora-Box

Moderator


Joined: 07 Sep 2006
Posts: 1529
Location: Andromeda Galaxy

PostPosted: Wed Dec 19, 2012 1:09 am    Post subject:
Reply with quote

Also do you have the option of running jobs in parallel?
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose

PostPosted: Wed Dec 19, 2012 1:19 am    Post subject: Re: Reply to: DFSORT is running longer due to huge Input vol
Reply with quote

V S Amarendra Reddy wrote:
This is total SYSOUT. The input/output are sequential files.

V S Amarendra Reddy,

If you need help, Please add //SORTDIAG DD DUMMY to your JCL and re-run the job and send us the complete sysout to DFSORT hot line: dfsort@us.ibm.com and our performance expert will look into it.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts High CPU consumption Job using IAM fi... aswinir JCL & VSAM 8 Thu Dec 01, 2016 8:28 pm
No new posts How to get the remainder and quotient... vnktrrd DFSORT/ICETOOL 2 Mon Oct 31, 2016 10:59 am
No new posts Problem in Running Query via JCL vickey_dw DB2 3 Tue Oct 18, 2016 11:11 pm
This topic is locked: you cannot edit posts or make replies. How to use 2 input files in control c... Gunapala CN DFSORT/ICETOOL 23 Thu Oct 13, 2016 3:42 pm
No new posts Change date (DD/MM/YY) in 2nd record ... uday kiran DFSORT/ICETOOL 12 Wed Sep 07, 2016 10:57 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us