IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Sorting Tape files efficiently than by the ordinary sort


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
raak

Active User


Joined: 23 May 2006
Posts: 166
Location: chennai

PostPosted: Wed Dec 13, 2006 6:32 pm
Reply with quote

Hi

i want to know whether there is any way in which a tape file can be sorted more efficiently than by the ordinary sort.

i have 10 daily jobs which have sort steps involving Tape files as input. it is consuming a lot of time as the input can have millions of records.

Is there any other efficient way in which the time consumption can be reduced.

I thought i had come across some topic like this. But a search didnt provide me with any useful results..


so hoping that someone will come up with an answer right away
Back to top
View user's profile Send private message
demora

New User


Joined: 12 Dec 2006
Posts: 4
Location: Cleveland, OH

PostPosted: Sat Dec 16, 2006 3:25 am
Reply with quote

Have you considered just copying the tapes to a temp data set (SORT FIELDS=COPY), sorting and then writing back to a tape? This way, the tapes are only accessed twice which should speed up the sort/process time.

That's about the only way I can think of right off to save processing time.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Sat Dec 16, 2006 4:08 am
Reply with quote

Hello,

If you can change the block sizes of the input and output tape files, make them as large as you can - this will reduce the number of physical i/o's which is where much of the tape time is spent.

Greatly increasing the region size may also help. (Frank?) The more core the sort has to use, the less sort i/o's may be needed. . . .

My own personal preference for very large sorts is to include several SORTWKnn DD statements.
Back to top
View user's profile Send private message
Frank Yaeger

DFSORT Developer


Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA

PostPosted: Sat Dec 16, 2006 4:25 am
Reply with quote

Quote:
Greatly increasing the region size may also help. (Frank?)


I can't comment on this based on the small amount of information given by the poster. I don't know which sort product is being used, I don't know anything specific about the job, and I don't know whether or not the statement about "consuming a lot of time" is true or not.
Back to top
View user's profile Send private message
MFRASHEED

Active User


Joined: 14 Jun 2005
Posts: 186
Location: USA

PostPosted: Sat Dec 16, 2006 6:36 am
Reply with quote

Also add buffers to Tape dataset (12 or 16). The DEFAULT BUFFER setting for a QSAM file is 5 which is bottle neck.

Code:


//TAPEDD1 DD  DSN=TAPE.DATASET(0),       
//             DISP=OLD,                                   
//             DCB=BUFNO=12             



And as Dick mentioned, change BLKSIZE. The BLOCKSIZE will be a multiple of the LRECL, up to 32760 for TAPE.

And here is quote from Sort manual(Read SORT Manual):

Quote:

Optimizing Tape Sort
Three factors are crucial to Tape Sort efficiency: a generous amount of intermediate storage,
a closely estimated input size value on the SORT or EXEC statement, and the freedom
to select the best sorting technique (BALN, OSCL or POLY) based on the nature of the
application and conditions at execution time. Accordingly, the number of SORTWK data
sets should be in excess of those suggested in the chart above, the BALN/OSCL/POLY
PARM should be omitted and an accurate SIZE or FILSZ estimate should be provided.


For more tips provide more details as Frank has asked for.
Back to top
View user's profile Send private message
Frank Yaeger

DFSORT Developer


Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA

PostPosted: Sat Dec 16, 2006 7:04 am
Reply with quote

MFRASHEED,

DFSORT ignores BUFNO since it does its own buffer optimization. I'd guess the same is true for Syncsort.

The discussion of "Optimizing Tape Sort" is I believe from the Syncsort manual. "Tape Sort" refers to using tapes as work data sets which is highly discouraged. The poster is "using tape files as input" which has nothing to do with "tape sort" unless he's also using tape work data sets in which case switching to DASD work data sets would speed things up considerably.
Back to top
View user's profile Send private message
MFRASHEED

Active User


Joined: 14 Jun 2005
Posts: 186
Location: USA

PostPosted: Mon Dec 18, 2006 9:23 pm
Reply with quote

Thanks Frank for correction.
Back to top
View user's profile Send private message
raak

Active User


Joined: 23 May 2006
Posts: 166
Location: chennai

PostPosted: Thu Jan 04, 2007 7:04 pm
Reply with quote

The sort product used in our shop is SYNCSORT.

The output dataset is defined like shown below.

SORTOUT DD DSN=XXXX.XXXX(+1),
UNIT=VTAPE,RETPD=10,
DCB=(MBB.MODEL,RECFM=VB,LRECL=16397,BLKSIZE=0),
DISP=(NEW,CATLG,DELETE)


what can be done to make this sort more efficiently???
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Thu Jan 04, 2007 8:57 pm
Reply with quote

Hello,

How many records are on the tape(s) and how long does it take to do the sort? Does the run time vary depending on the time of day? How many files are input to the sort? If there is concatenated input and the data is already in the proper sequence, use MERGE rather than SORT - it is exponentially faster. The sort may be performing well, but due to time constraints it needs to be sped up.

Given that you have specified VTAPE, it is most likely not "really" a physical tape but rather a virtual tape. Usually virtual tape outperforms physical tape.

Look at (or post here) your sort control statement(s). If you have not specified an estimated number of records to sort, do so.

If you have a BIG sort (i.e. a billion records) and if your site uses high-density tapes, you might run a timing test sending the output to one of these tapes.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM

 


Similar Topics
Topic Forum Replies
No new posts How to split large record length file... DFSORT/ICETOOL 8
No new posts Write line by line from two files DFSORT/ICETOOL 7
No new posts Compare only first records of the fil... SYNCSORT 7
No new posts JCL sort card - get first day and las... JCL & VSAM 9
No new posts Sort First/last record of a subset th... DFSORT/ICETOOL 7
Search our Forums:

Back to Top