I have a job that is taking a long time to execute in production. Last exec took 177 minutes to execute, and total CPU Time= 5.15
We have identified the problem steps in some sorts, which allocate huge files in tapes.
For standard, the JCL downloaded the tape to disk (SYSDA) and then, merged it with a sort with new data, and deleted/recreated the tape. This last step was taking a long time (Clock Time: 01:16:32 , CPU TIME 165.59S)
We discarded that and decided to simply copy with an ICEGENER the new records to the TAPE with disp=MOD, so that should reduce the exec time drastically.
However, I've been told by A support guy to add the BUFNO parameter to the files, however, I've been reading through the forum that the BUFNO parameter is not taken in consideration when the program processing the file is DFSORT.
We believe that this should decrease the processing time, since there won't be anymore sorting needed, nor delete/create of the tape before processing.
So my questions are:
BUFNO won't do anything to improve performance on that step, right? and, can anything else be done besides changing the sorting and recreation of the tape to a copy, as in the last example?
are You aware of the dangers of using disp=mod... you might be clobbering things
why not concatenate the tape and dasd datasets and create a new tape? but,
investigate how ICEGENER I/O behaves for concatenations of datasets on different device types
maybe a MERGE if the datasets are properly sorted might give better results ( from an I/O perspective).
( for MERGE the I/O technique used might be different and optimized for each input involved )
beware this is not a solution, but a hint on few issues to meditate about.
Frank or Kolusu are certainly the ones to speak the last word about it
That's exactly what we did before, just with a Sort and it took quite some time to execute. We tried changing the creation of a new tape with ICEGENER and the times were still elevated, that's when we decided to use an disp=MOD
As for the dangers of it, What i know so far is that if the device runs out of space, the file could be damaged beyond repair. However, some STORAGE support guys told me that the "Space is almost infinite", not the best technical or reassuring answer I could get, but that's what they've told me
We have made some test with the disp=MOD and have been succesful so far. Around 120 executions in production. Here is one of the sysouts of a step with an ICEGENER And disp=MOD
Code:
BLOCKSET COPY TECHNIQUE SELECTED
VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
- CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R5 - 10:24 ON WED SEP 09, 2009
OPTION COPY,MSGDDN=SYSPRINT,SORTIN=SYSUT1,SORTOUT=SYSUT2
RECORD TYPE IS F - DATA STARTS IN POSITION 1
C5-K90013 C6-K90013 C7-K90000 C8-K42135 E9-K90013 C9-BASE E5-K44563 E7-K44563
ICEAM2 ENVIRONMENT IN EFFECT - ICEAM2 INSTALLATION MODULE SELECTED
HAMDMHI2.PASO040 . , INPUT LRECL = 300, BLKSIZE = 27900, TYPE = FB
MAIN STORAGE = (MAX,6291456,6291456)
MAIN STORAGE ABOVE 16MB = (6234096,6234096)
OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VS
OPTIONS: SIZE=6291456,MAXLIM=1048576,MINLIM=450560,EQUALS=N,LIST=Y,ERET=RC16 ,MS
OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=
OPTIONS: RESALL=8192,RESINV=32768,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,STIM
OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=0
OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDD
OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE
OPTIONS: NULLOUT=RC0
EXCP ACCESS METHOD USED FOR SYSUT2
EXCP ACCESS METHOD USED FOR SYSUT1
EF-K10929 F0-K30362 E8-K44563
OUTPUT LRECL = 300, BLKSIZE = 32700, TYPE = FB
INSERT 0, DELETE 0
RECORDS - IN: 3292704, OUT: 3292704
END OF DFSORT
The execution times were acceptable. Around ELAPSED-TIME: 00:03:39 and CPU-TIME: 2.17S.
Right now, we decided that the records aren't needed in order, ergo the use of the append option with disp=mod and the icegener.
That said, is there anything additional I should be aware of disp=MOD? I tend to aboid using it for large files, but in this case, that's the only approach we have found so far. Could you please elaborate a little bit more when you say:
That's what i heard before. Well, we made this process so that the reocovery of a damaged file could be done from a DB2 table that is loaded before with the data in the file. We are appending the records to it as to have an historical file that won't be deleted. That historical file has accounting movements of the current month, so when a new month arrives, a new file is created. The table, as is loaded and has only the last 3 months data on it, can be downloaded and used to recreate the file if something goes wrong.
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
If you don't need the records in order, then you should be doing a COPY, not a SORT. COPY is generally much faster then SORT.
You can do the COPY with the PGM=SORT job by using OPTION COPY. You can also do it with PGM=ICEGENER. BUFNO will have no effect if DFSORT is used (DFSORT does its own buffering).
I don't understand what the use of DISP=MOD vs DISP=NEW has to do with the runtime unless it's some kind of special processing done for tapes in your installation. AFAIK, using DISP=NEW for a tape does not normally add significant runtime for a job.
If you want, add the following to the job:
//SORTDIAG DD DUMMY
to get diagnostic messages. Then run the job with DISP=NEW and DISP=MOD and send me (yaeger@us.ibm.com) the JES messages for both runs and I'll take a look.
Maybe I wasn't clear enough when i addressed the issue about the DISP=NEW Vs DISP=MOD, I'm sorry for the confussion. I'll explain what the job used to do in detail:
First, We have a tape which will receive records in an incrementally. First day of the month the tape is create and receives around 10 million records, and everyday, more records are added, from 6 million up to 12 million in a given day
With that we forgot about the sort steps and also the copy to disk and later upload to the tape. That's what I Meant when we changed DISP=NEW to DISP=MOD
Today it was executed like this and it took only around 15 minutes CLOCK TIME to execute the job in total. Previously it took around 3 hours. Now the question is: It's too risky to copy the records to the TAPE like this? AS I stated early, I'm no friend of DISP=MOD, but I think this case justifies it
Well, that sums it up.
I'll Add the //SORTDIAG DD DUMMY card to the job and test it and then I'll send you the results.
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
If you just want to keep adding the unsorted new records to the end of the tape, then this should work, although I won't comment on the dangers of DISP=MOD since I don't know much about that. My only comment about DISP=MOD had to do with performance vs DISP=NEW, but now that you've explained what you were doing before and what you're doing now, I see that DISP=MOD vs DISP=NEW had nothing to do with performance.
I'm sure you realize that what you're doing with the one ICEGENER step is NOT EQUIVALENT to what you were doing before. The job you used before resulted in all of the records (old and new) being sorted (that's why it took longer). The job you're using now results in appending new records without sorting them which requires much less processing.
Quote:
I'll Add the //SORTDIAG DD DUMMY card to the job and test it and then I'll send you the results.
If you just want to keep adding the unsorted new records to the end of the tape, then this should work, although I won't comment on the dangers of DISP=MOD since I don't know much about that. My only comment about DISP=MOD had to do with performance vs DISP=NEW, but now that you've explained what you were doing before and what you're doing now, I see that DISP=MOD vs DISP=NEW had nothing to do with performance.
I'm sure you realize that what you're doing with the one ICEGENER step is NOT EQUIVALENT to what you were doing before. The job you used before resulted in all of the records (old and new) being sorted (that's why it took longer). The job you're using now results in appending new records without sorting them which requires much less processing.
Quote:
I'll Add the //SORTDIAG DD DUMMY card to the job and test it and then I'll send you the results.
Given what you've said, that's not necessary.
Certainly Frank. We discussed the need to have the records sorted in the tape and realized it was not an issue since it is only a historical file. After all, maybe the records will be needed on a different sorting order once them have to be exploited.
Once we reached that conclusion, was the moment when we decided to drop the sort and resort to the ICEGENER. Since there is no sort process involved (that is, sorting the records of both files) we would be saving processing time at all
Ok, then I won't send you the result of the //SORTDIAG, however, i had it executed tonight and I'll take a look at it just for curiousity to see what relevant or important info it could provide me. Maybe I'll learn something from it.
