Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

EXEC TIMES - Optimization

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL
View previous topic :: :: View next topic  
Author Message
ojdiaz

New User


Joined: 19 Nov 2008
Posts: 90
Location: Spain

PostPosted: Wed Sep 09, 2009 1:33 pm    Post subject: EXEC TIMES - Optimization
Reply with quote

Hello

I have a job that is taking a long time to execute in production. Last exec took 177 minutes to execute, and total CPU Time= 5.15

We have identified the problem steps in some sorts, which allocate huge files in tapes.

For standard, the JCL downloaded the tape to disk (SYSDA) and then, merged it with a sort with new data, and deleted/recreated the tape. This last step was taking a long time (Clock Time: 01:16:32 , CPU TIME 165.59S)

We discarded that and decided to simply copy with an ICEGENER the new records to the TAPE with disp=MOD, so that should reduce the exec time drastically.

However, I've been told by A support guy to add the BUFNO parameter to the files, however, I've been reading through the forum that the BUFNO parameter is not taken in consideration when the program processing the file is DFSORT.

Here is the Sort step and it's stats:

Code:

//PASO19B EXEC PGM=SORT                                         
//SORTIN   DD DSN=INFILEA,DISP=SHR
//         DD DSN=INFILEB,DISP=SHR
//SORTOUT  DD DSN=TAPEOUT,           
//            DISP=(NEW,CATLG,DELETE),                           
//            DCB=(RECFM=FBM,LRECL=300,BLKSIZE=0),               
//            UNIT=VTSA                                         
//SYSPRINT DD SYSOUT=*                                           
//SYSOUT   DD SYSOUT=*                                           
//SYSIN    DD *                                                 
  SORT FIELDS=(1,4,CH,A,
               8,4,CH,A,
               204,15,CH,A,
               95,4,CH,A,   
               263,3,CH,A, 
               64,1,CH,A),FORMAT=BI
  OPTION DYNALLOC=(,10)                                         


And here is the Sysout:

Code:

RECORD TYPE IS F - DATA STARTS IN POSITION 1                                   
C5-K90013 C6-K90013 C7-K90000 C8-K42135 E4-K90007 C9-BASE   E5-K44563 E6-K34782
ICEAM1 ENVIRONMENT IN EFFECT - ICEAM1 INSTALLATION MODULE SELECTED             
HAMDM670.PASO19B .        , INPUT LRECL = 300, BLKSIZE = 27900, TYPE = FB       
MAIN STORAGE = (MAX,62285980,62285980)                                         
MAIN STORAGE ABOVE 16MB = (62090540,62090540)                                   
OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
OPTIONS: SIZE=62285980,MAXLIM=2097152,MINLIM=450560,EQUALS=Y,LIST=Y,ERET=ABEND,MSGDDN=SYSOUT
OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO   ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(SYSDA   ,010),ABCODE=MSG
OPTIONS: RESALL=8192,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,STIMER=Y,COBEXIT=COB2
OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DSA=64 
OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
OPTIONS: NULLOUT=RC0                                                           
EXCP ACCESS METHOD USED FOR SORTOUT                                             
EXCP ACCESS METHOD USED FOR SORTIN                                             
DC 26392535100 TC 0 CS DSVVV KSZ 35 VSZ 35                                     
FSZ=87975117 RC  IGN=0 E  AVG=304 0  WSP=34736425 C  DYN=551568 53216           
DE-K24705 D5-K24705 D3-K24705 ED-K24705 E8-K44563                               
OUTPUT LRECL = 300, BLKSIZE = 32700, TYPE = FB   (SDB)                         
INSERT 0, DELETE 0                                                             
RECORDS - IN: 87975076, OUT: 87975076                                           
NUMBER OF BYTES SORTED: 26392522800                                             
TOTAL WORK DATA SET TRACKS ALLOCATED: 551640 , TRACKS USED: 487095             
MEMORY OBJECT STORAGE USED = 0M BYTES                                           
HIPERSPACE STORAGE USED = 2506376K BYTES                                       
DATA SPACE STORAGE USED = 0K BYTES                                             


As I was saying, we changed that step to do something like:

Code:

//PASO19B EXEC PGM=ICEGENER                                       
//SYSUT1   DD DSN=INFILEA,DISP=SHR
//SYSUT2   DD DSN=TAPEOUT,DISP=MOD
//SYSPRINT DD SYSOUT=*                                             
//SYSOUT   DD SYSOUT=*                                             
//SYSIN    DD DUMMY               


We believe that this should decrease the processing time, since there won't be anymore sorting needed, nor delete/create of the tape before processing.

So my questions are:

BUFNO won't do anything to improve performance on that step, right? and, can anything else be done besides changing the sorting and recreation of the tape to a copy, as in the last example?

As usual, thanks a lot in advance

Best Regards

Oliver
Back to top
View user's profile Send private message

enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10254
Location: italy

PostPosted: Wed Sep 09, 2009 1:54 pm    Post subject: Reply to: EXEC TIMES - Optimization
Reply with quote

are You aware of the dangers of using disp=mod... you might be clobbering things

why not concatenate the tape and dasd datasets and create a new tape? but,
investigate how ICEGENER I/O behaves for concatenations of datasets on different device types

maybe a MERGE if the datasets are properly sorted might give better results ( from an I/O perspective).
( for MERGE the I/O technique used might be different and optimized for each input involved )

beware this is not a solution, but a hint on few issues to meditate about.

Frank or Kolusu are certainly the ones to speak the last word about it
Back to top
View user's profile Send private message
ojdiaz

New User


Joined: 19 Nov 2008
Posts: 90
Location: Spain

PostPosted: Wed Sep 09, 2009 2:12 pm    Post subject:
Reply with quote

Hello Enrico

Thanks for ur support.

That's exactly what we did before, just with a Sort and it took quite some time to execute. We tried changing the creation of a new tape with ICEGENER and the times were still elevated, that's when we decided to use an disp=MOD

As for the dangers of it, What i know so far is that if the device runs out of space, the file could be damaged beyond repair. However, some STORAGE support guys told me that the "Space is almost infinite", not the best technical or reassuring answer I could get, but that's what they've told me

We have made some test with the disp=MOD and have been succesful so far. Around 120 executions in production. Here is one of the sysouts of a step with an ICEGENER And disp=MOD

Code:

BLOCKSET     COPY  TECHNIQUE SELECTED                                           
VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE   
- CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R5 - 10:24 ON WED SEP 09, 2009
OPTION COPY,MSGDDN=SYSPRINT,SORTIN=SYSUT1,SORTOUT=SYSUT2                       
RECORD TYPE IS F - DATA STARTS IN POSITION 1                                   
C5-K90013 C6-K90013 C7-K90000 C8-K42135 E9-K90013 C9-BASE   E5-K44563 E7-K44563
ICEAM2 ENVIRONMENT IN EFFECT - ICEAM2 INSTALLATION MODULE SELECTED             
HAMDMHI2.PASO040 .        , INPUT LRECL = 300, BLKSIZE = 27900, TYPE = FB       
MAIN STORAGE = (MAX,6291456,6291456)                                           
MAIN STORAGE ABOVE 16MB = (6234096,6234096)                                     
OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VS
OPTIONS: SIZE=6291456,MAXLIM=1048576,MINLIM=450560,EQUALS=N,LIST=Y,ERET=RC16 ,MS
OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO   ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=
OPTIONS: RESALL=8192,RESINV=32768,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,STIM
OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=0   
OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXITCK=S,PARMDD
OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE
OPTIONS: NULLOUT=RC0                                                           
EXCP ACCESS METHOD USED FOR SYSUT2                                             
EXCP ACCESS METHOD USED FOR SYSUT1                                             
EF-K10929 F0-K30362 E8-K44563                                                   
OUTPUT LRECL = 300, BLKSIZE = 32700, TYPE = FB                                 
INSERT 0, DELETE 0                                                             
RECORDS - IN: 3292704, OUT: 3292704                                             
END OF DFSORT                                                                   


The execution times were acceptable. Around ELAPSED-TIME: 00:03:39 and CPU-TIME: 2.17S.

Right now, we decided that the records aren't needed in order, ergo the use of the append option with disp=mod and the icegener.

That said, is there anything additional I should be aware of disp=MOD? I tend to aboid using it for large files, but in this case, that's the only approach we have found so far. Could you please elaborate a little bit more when you say:

Quote:

you might be clobbering things


What does that means exactly?

Thanks Again

Oliver
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10254
Location: italy

PostPosted: Wed Sep 09, 2009 2:18 pm    Post subject: Reply to: EXEC TIMES - Optimization
Reply with quote

that <things> refers to the output dataset ,

if something happens when appending the output dataset might be logically unusable

the dasd dataset only partially appended, and the cleanup might be tricky
Back to top
View user's profile Send private message
ojdiaz

New User


Joined: 19 Nov 2008
Posts: 90
Location: Spain

PostPosted: Wed Sep 09, 2009 2:55 pm    Post subject:
Reply with quote

That's what i heard before. Well, we made this process so that the reocovery of a damaged file could be done from a DB2 table that is loaded before with the data in the file. We are appending the records to it as to have an historical file that won't be deleted. That historical file has accounting movements of the current month, so when a new month arrives, a new file is created. The table, as is loaded and has only the last 3 months data on it, can be downloaded and used to recreate the file if something goes wrong.

Thanks for ur support

Oliver
Back to top
View user's profile Send private message
Frank Yaeger

DFSORT Moderator


Joined: 15 Feb 2005
Posts: 7130
Location: San Jose, CA

PostPosted: Wed Sep 09, 2009 9:35 pm    Post subject:
Reply with quote

If you don't need the records in order, then you should be doing a COPY, not a SORT. COPY is generally much faster then SORT.

You can do the COPY with the PGM=SORT job by using OPTION COPY. You can also do it with PGM=ICEGENER. BUFNO will have no effect if DFSORT is used (DFSORT does its own buffering).

I don't understand what the use of DISP=MOD vs DISP=NEW has to do with the runtime unless it's some kind of special processing done for tapes in your installation. AFAIK, using DISP=NEW for a tape does not normally add significant runtime for a job.

If you want, add the following to the job:

//SORTDIAG DD DUMMY

to get diagnostic messages. Then run the job with DISP=NEW and DISP=MOD and send me (yaeger@us.ibm.com) the JES messages for both runs and I'll take a look.
Back to top
View user's profile Send private message
ojdiaz

New User


Joined: 19 Nov 2008
Posts: 90
Location: Spain

PostPosted: Thu Sep 10, 2009 1:49 pm    Post subject:
Reply with quote

Maybe I wasn't clear enough when i addressed the issue about the DISP=NEW Vs DISP=MOD, I'm sorry for the confussion. I'll explain what the job used to do in detail:

First, We have a tape which will receive records in an incrementally. First day of the month the tape is create and receives around 10 million records, and everyday, more records are added, from 6 million up to 12 million in a given day

The job does the following.

Takes the Tape and copies it to a file in disk:

Code:
//PASO13A EXEC PGM=ICEGENER                               
//SYSUT1   DD DSN=TAPE,DISP=SHR
//SYSUT2   DD DSN=DISK_BACKUP,
//            DISP=(NEW,CATLG,DELETE),                   
//            DCB=(RECFM=FBM,LRECL=300,BLKSIZE=0),       
//            SPACE=(CYL,(500,50),RLSE),UNIT=SYSDA       
//SYSPRINT DD SYSOUT=*                                   
//SYSIN    DD DUMMY                                       

Deletes the TAPE

Code:
//PASO14A  EXEC PGM=IEFBR14                                             
//SYSPRINT DD SYSOUT=*                                                 
//DD01     DD DSN=TAPE,                     
//            DISP=(MOD,DELETE,DELETE),UNIT=VTS,                       
//            SPACE=(TRK,(0)),DCB=(BLKSIZE=0)                           

Finally, it creates the tape again, merging two files, the disk backup of the tape records and that day records:

Code:
//PASO19B EXEC PGM=SORT                                                 
//SORTIN   DD DSN=DISK_BACKUP,DISP=SHR     
//         DD DSN=DAILY_MOVEMENTS,DISP=SHR   --> AROUND 10 MILLION RECORDS
//SORTOUT  DD DSN=TAPE,             
//            DISP=(NEW,CATLG,DELETE),                                 
//            DCB=(RECFM=FBM,LRECL=300,BLKSIZE=0),                     
//            UNIT=VTS                                                 
//SYSPRINT DD SYSOUT=*                                                 
//SYSOUT   DD SYSOUT=*                                                 
//SYSIN    DD *                                                         
  SORT FIELDS=(1,4,CH,A,                 ENTIDAD                       
               8,4,CH,A,                 ANIO                           
               204,15,CH,A,              CUENTA CONTABLE               
               95,4,CH,A,                CENTRO DESTINO                 
               263,3,CH,A,               MONEDA                         
               64,1,CH,A),FORMAT=BI      TIPO DE MONEDA                 
  OPTION DYNALLOC=(,%%AREAS)                                           


The steps PASO13A and PASO19B are the ones taking a long time.

It was made that way in order to prevent damage to the historical tapes when adding records and the file is being used in DISP=MOD

However, the optimization done was this:

Eliminate the Disk_Backup and delete steps (PASO13A and PASO19B), and add directly the records to the tape, already create, by means of an ICEGENER:
Code:

//PASO19B EXEC PGM=ICEGENER                                     
//SYSUT1   DD DSN=DAILY_MOVEMENTS,DISP=SHR
//SYSUT2   DD DSN=TAPE,DISP=MOD   
//SYSPRINT DD SYSOUT=*                                           
//SYSOUT   DD SYSOUT=*                                           
//SYSIN    DD DUMMY                                             


With that we forgot about the sort steps and also the copy to disk and later upload to the tape. That's what I Meant when we changed DISP=NEW to DISP=MOD

Today it was executed like this and it took only around 15 minutes CLOCK TIME to execute the job in total. Previously it took around 3 hours. Now the question is: It's too risky to copy the records to the TAPE like this? AS I stated early, I'm no friend of DISP=MOD, but I think this case justifies it

Well, that sums it up.

I'll Add the //SORTDIAG DD DUMMY card to the job and test it and then I'll send you the results.

Thanks a lot for your patience and great support

Best Regards

Oliver Díaz
Back to top
View user's profile Send private message
Frank Yaeger

DFSORT Moderator


Joined: 15 Feb 2005
Posts: 7130
Location: San Jose, CA

PostPosted: Thu Sep 10, 2009 8:59 pm    Post subject:
Reply with quote

If you just want to keep adding the unsorted new records to the end of the tape, then this should work, although I won't comment on the dangers of DISP=MOD since I don't know much about that. My only comment about DISP=MOD had to do with performance vs DISP=NEW, but now that you've explained what you were doing before and what you're doing now, I see that DISP=MOD vs DISP=NEW had nothing to do with performance.

I'm sure you realize that what you're doing with the one ICEGENER step is NOT EQUIVALENT to what you were doing before. The job you used before resulted in all of the records (old and new) being sorted (that's why it took longer). The job you're using now results in appending new records without sorting them which requires much less processing.

Quote:
I'll Add the //SORTDIAG DD DUMMY card to the job and test it and then I'll send you the results.


Given what you've said, that's not necessary.
Back to top
View user's profile Send private message
ojdiaz

New User


Joined: 19 Nov 2008
Posts: 90
Location: Spain

PostPosted: Thu Sep 10, 2009 10:50 pm    Post subject:
Reply with quote

Frank Yaeger wrote:
If you just want to keep adding the unsorted new records to the end of the tape, then this should work, although I won't comment on the dangers of DISP=MOD since I don't know much about that. My only comment about DISP=MOD had to do with performance vs DISP=NEW, but now that you've explained what you were doing before and what you're doing now, I see that DISP=MOD vs DISP=NEW had nothing to do with performance.

I'm sure you realize that what you're doing with the one ICEGENER step is NOT EQUIVALENT to what you were doing before. The job you used before resulted in all of the records (old and new) being sorted (that's why it took longer). The job you're using now results in appending new records without sorting them which requires much less processing.

Quote:
I'll Add the //SORTDIAG DD DUMMY card to the job and test it and then I'll send you the results.


Given what you've said, that's not necessary.


Certainly Frank. We discussed the need to have the records sorted in the tape and realized it was not an issue since it is only a historical file. After all, maybe the records will be needed on a different sorting order once them have to be exploited.

Once we reached that conclusion, was the moment when we decided to drop the sort and resort to the ICEGENER. Since there is no sort process involved (that is, sorting the records of both files) we would be saving processing time at all

Ok, then I won't send you the result of the //SORTDIAG, however, i had it executed tonight and I'll take a look at it just for curiousity to see what relevant or important info it could provide me. Maybe I'll learn something from it.

Anyway, thanks again for your support

Best Regards,

Oliver
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Running an REXX EXEC implicitly sho7hills CLIST & REXX 3 Thu Apr 20, 2017 8:14 am
No new posts Strange EXEC function call in z/VM Willy Jensen CLIST & REXX 3 Wed Oct 05, 2016 2:07 pm
No new posts hot to get details when "EXEC CI... Andi1982 CICS 11 Tue Sep 20, 2016 5:01 pm
No new posts How to calculate an specific transact... lind sh CICS 2 Tue Sep 13, 2016 9:20 pm
No new posts SQL Query optimization. arunsoods DB2 12 Sun Sep 04, 2016 6:27 am


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us