View previous topic :: View next topic
|
Author |
Message |
raak
Active User
Joined: 23 May 2006 Posts: 166 Location: chennai
|
|
|
|
Hi
i want to know whether there is any way in which a tape file can be sorted more efficiently than by the ordinary sort.
i have 10 daily jobs which have sort steps involving Tape files as input. it is consuming a lot of time as the input can have millions of records.
Is there any other efficient way in which the time consumption can be reduced.
I thought i had come across some topic like this. But a search didnt provide me with any useful results..
so hoping that someone will come up with an answer right away |
|
Back to top |
|
|
demora
New User
Joined: 12 Dec 2006 Posts: 4 Location: Cleveland, OH
|
|
|
|
Have you considered just copying the tapes to a temp data set (SORT FIELDS=COPY), sorting and then writing back to a tape? This way, the tapes are only accessed twice which should speed up the sort/process time.
That's about the only way I can think of right off to save processing time. |
|
Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
|
|
|
|
Hello,
If you can change the block sizes of the input and output tape files, make them as large as you can - this will reduce the number of physical i/o's which is where much of the tape time is spent.
Greatly increasing the region size may also help. (Frank?) The more core the sort has to use, the less sort i/o's may be needed. . . .
My own personal preference for very large sorts is to include several SORTWKnn DD statements. |
|
Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
|
|
|
|
Quote: |
Greatly increasing the region size may also help. (Frank?) |
I can't comment on this based on the small amount of information given by the poster. I don't know which sort product is being used, I don't know anything specific about the job, and I don't know whether or not the statement about "consuming a lot of time" is true or not. |
|
Back to top |
|
|
MFRASHEED
Active User
Joined: 14 Jun 2005 Posts: 186 Location: USA
|
|
|
|
Also add buffers to Tape dataset (12 or 16). The DEFAULT BUFFER setting for a QSAM file is 5 which is bottle neck.
Code: |
//TAPEDD1 DD DSN=TAPE.DATASET(0),
// DISP=OLD,
// DCB=BUFNO=12
|
And as Dick mentioned, change BLKSIZE. The BLOCKSIZE will be a multiple of the LRECL, up to 32760 for TAPE.
And here is quote from Sort manual(Read SORT Manual):
Quote: |
Optimizing Tape Sort
Three factors are crucial to Tape Sort efficiency: a generous amount of intermediate storage,
a closely estimated input size value on the SORT or EXEC statement, and the freedom
to select the best sorting technique (BALN, OSCL or POLY) based on the nature of the
application and conditions at execution time. Accordingly, the number of SORTWK data
sets should be in excess of those suggested in the chart above, the BALN/OSCL/POLY
PARM should be omitted and an accurate SIZE or FILSZ estimate should be provided.
|
For more tips provide more details as Frank has asked for. |
|
Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
|
|
|
|
MFRASHEED,
DFSORT ignores BUFNO since it does its own buffer optimization. I'd guess the same is true for Syncsort.
The discussion of "Optimizing Tape Sort" is I believe from the Syncsort manual. "Tape Sort" refers to using tapes as work data sets which is highly discouraged. The poster is "using tape files as input" which has nothing to do with "tape sort" unless he's also using tape work data sets in which case switching to DASD work data sets would speed things up considerably. |
|
Back to top |
|
|
MFRASHEED
Active User
Joined: 14 Jun 2005 Posts: 186 Location: USA
|
|
|
|
Thanks Frank for correction. |
|
Back to top |
|
|
raak
Active User
Joined: 23 May 2006 Posts: 166 Location: chennai
|
|
|
|
The sort product used in our shop is SYNCSORT.
The output dataset is defined like shown below.
SORTOUT DD DSN=XXXX.XXXX(+1),
UNIT=VTAPE,RETPD=10,
DCB=(MBB.MODEL,RECFM=VB,LRECL=16397,BLKSIZE=0),
DISP=(NEW,CATLG,DELETE)
what can be done to make this sort more efficiently??? |
|
Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
|
|
|
|
Hello,
How many records are on the tape(s) and how long does it take to do the sort? Does the run time vary depending on the time of day? How many files are input to the sort? If there is concatenated input and the data is already in the proper sequence, use MERGE rather than SORT - it is exponentially faster. The sort may be performing well, but due to time constraints it needs to be sped up.
Given that you have specified VTAPE, it is most likely not "really" a physical tape but rather a virtual tape. Usually virtual tape outperforms physical tape.
Look at (or post here) your sort control statement(s). If you have not specified an estimated number of records to sort, do so.
If you have a BIG sort (i.e. a billion records) and if your site uses high-density tapes, you might run a timing test sending the output to one of these tapes. |
|
Back to top |
|
|
|