IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

how to reduce the run time for the below sort
IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic Quick References
View previous topic :: View next topic  
Author Message
maddukurivasu
Warnings : 1
New User


Joined: 14 Apr 2006
Posts: 4
PostPosted: Fri Feb 17, 2012 2:07 pm
 Reply with quote
My input file having 5 million records it takes 2 hours time to complete the job using below sort. please any one help me to reduce the time if any other option.

Code:
  SORT FIELDS=(19,15,ZD,A,715,2,ZD,A)
  OUTREC IFTHEN=(WHEN=INIT,
          BUILD=(1:1,714)),
       IFTHEN=(WHEN=(1,3,CH,EQ,C'HDR'),
          BUILD=(1:1,18,19:34,681,700:15X)),
       IFTHEN=(WHEN=(1,3,CH,EQ,C'TRL'),
          BUILD=(1:1,18,19:34,681,700:15X)),
       IFTHEN=(WHEN=(1,3,CH,NE,C'HDR',AND,1,3,CH,NE,C'TRL'),
        BUILD=(1:1,3,4:SEQNUM,15,ZD,START=1,19:19,696))
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix
PostPosted: Fri Feb 17, 2012 2:15 pm
 Reply with quote
Can you confirm which sort product you are using please?

Also LRECL and RECFM.

You are throwing around some large chunks of data twice per record. If you are able to describe your processing exactly, there may be some possibilities to reduce things, using OVERLAY, for instance.
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10873
Location: italy
PostPosted: Fri Feb 17, 2012 2:27 pm
 Reply with quote
premise ... I an not a sort expert, but usually when You are able to get a result using <less> statements it might give some advantages
no reason to use more statements than needed
why not simplify the compare logic...
You are in an IF THEN / ELSE situation
if You check for HDR and TRL for equality, no reason to check also for inequality

try something along the lines of
Code:
  SORT FIELDS=(19,15,ZD,A,715,2,ZD,A)
  OUTREC IFTHEN=(WHEN=(1,3,CH,EQ,C'HDR',OR,1,3,CH,EQ,C'TRL'),
          BUILD=(1:1,18,19:34,681,700:15X)),
       IFTHEN=(WHEN=NONE,
        BUILD=(1:1,3,4:SEQNUM,15,ZD,START=1,19:19,696))


checked with a simpler record/key setup, but the results are just the same
Back to top
View user's profile Send private message
maddukurivasu
Warnings : 1
New User


Joined: 14 Apr 2006
Posts: 4
PostPosted: Fri Feb 17, 2012 2:27 pm
 Reply with quote
Input file LRECL= 716, Output file LRECL=714,

RECFM= FB for both files,
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix
PostPosted: Sat Feb 18, 2012 1:00 am
 Reply with quote
enrico-sorichetti wrote:
[...]
Code:
  SORT FIELDS=(19,15,ZD,A,715,2,ZD,A)
  OUTREC IFTHEN=(WHEN=(1,3,CH,EQ,C'HDR',OR,1,3,CH,EQ,C'TRL'),
          BUILD=(1:1,18,19:34,681,700:15X)),
       IFTHEN=(WHEN=NONE,
        BUILD=(1:1,3,4:SEQNUM,15,ZD,START=1,19:19,696))


checked with a simpler record/key setup, but the results are just the same


maddukurivasu, this looks like a good start from enrico. Any comment from you? Any answer to my earlier request?

The above still has 19:19,696 - a 696-byte move of data which is already there, as far as I can tell (and without any help from you). If you feel you have to do the build just to drop off the last two bytes of the record (don't know, because you haven't said), you could try this in place of the build.
Code:
  OVERLAY=(1:1,3,4:SEQNUM,15,ZD,START=1))


That hasn't dealt with the record-length. So start with

Code:
  OUTREC IFOUTLEN=714,


If that is good for you, you will have removed two 700 byte moves per 5,000,000 records. That should be noticeable.

If you feel it still "too slow" have a look at your datasets: blocking; buffers; file types. Also consult your sort manual (we are assuming Syncsort from where you have posted) and look for information on performance/tuning and see if that covers anything you are doing.
Back to top
View user's profile Send private message
don.leahy

Active Member


Joined: 06 Jul 2010
Posts: 765
Location: Whitby, ON, Canada
PostPosted: Sat Feb 18, 2012 1:34 am
 Reply with quote
Are you sure that the OUTREC is the problem?

How long does the SORT step take to run if you take out the OUTREC entirely?
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia
PostPosted: Sat Feb 18, 2012 4:55 am
 Reply with quote
Hi,

I generated 1700000 records with HDR in pos 1-3
1700000 records with TRL in pos 1-3
1700000 records without either HDR or TRL in pos 1-3.

A total of 510000 records.

I ran the code provided by the OP and here are the results
Code:
TOTAL TCB CPU TIME=   .46  TOTAL ELAPSED TIME=   2.9


ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAM
ICE235I 0 OPTIONS: NULLOUT=RC0                                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN                                   
ICE750I 0 DC 3651621480 TC 0 CS DSVOO KSZ 21 VSZ 21                           
ICE752I 0 FSZ=5100030 RC  IGN=0 E  AVG=740 0  WSP=4901806 C  DYN=46340 53216   
ICE751I 1 DE-K61785 D5-K62201 D3-BASE   D7-BASE   E8-K61438                   
ICE090I 0 OUTPUT LRECL = 714, BLKSIZE = 27846, TYPE = FB   (SDB)               
ICE171I 0 SORTOUT LRECL OF 714 IS DIFFERENT FROM SORTIN(NN) LRECL OF 716 - RC=0
ICE055I 0 INSERT 0, DELETE 0                                                   
ICE054I 0 RECORDS - IN: 5100000, OUT: 5100000                                 
ICE134I 0 NUMBER OF BYTES SORTED: 3651600000                                   
ICE253I 0 RECORDS SORTED - PROCESSED: 5100000, EXPECTED: 5100030               
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 46350 , TRACKS USED: 31080     
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES                               
ICE180I 0 HIPERSPACE STORAGE USED = 2231132K BYTES                             
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES                                   
ICE052I 0 END OF DFSORT                                                       



This is using DFSORT but can't imagine the difference would be so dramatic betwwen the 2 products.


Is the input on tape or disk and whatt is the blocksize for both input and output ?

Gerry
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix
PostPosted: Sat Feb 18, 2012 5:29 am
 Reply with quote
So, this is good. Even if we cut the OUTREC processing by 90%, we won't notice :-)

I suspect a COPY would give very similar results, as you've probably not generated any "random" values for the keys, but it certainly serves our purpose. The OUTREC is not the problem, good spot Don and Gerry.

We need the JCL and the output messages. Also interesting to know how that 15-digit key is made up. Is it several things? If so, how many, and how much do the values vary?
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia
PostPosted: Sat Feb 18, 2012 10:32 am
 Reply with quote
Hi Bill,

actually I had generated a sequence number as such

Code:
  OPTION COPY                                                         
  OUTFIL REPEAT=1700000,OVERLAY=(19:SEQNUM,15,ZD,START=0,             
                715:SEQNUM,02,ZD,START=0)                             


This time I decided to vary the sort to
Code:
  SORT FIELDS=(30,4,ZD,D,22,4,ZD,A,26,4,ZD,D,19,3,ZD,A,715,2,ZD,A)
  OUTREC IFTHEN=(WHEN=INIT,                                       
          BUILD=(1:1,714)),                                       
       IFTHEN=(WHEN=(1,3,CH,EQ,C'HDR'),                           
          BUILD=(1:1,18,19:34,681,700:15X)),                       
       IFTHEN=(WHEN=(1,3,CH,EQ,C'TRL'),                           
          BUILD=(1:1,18,19:34,681,700:15X)),                       
       IFTHEN=(WHEN=(1,3,CH,NE,C'HDR',AND,1,3,CH,NE,C'TRL'),       
        BUILD=(1:1,3,4:SEQNUM,15,ZD,START=1,19:19,696))           



The result
Code:
 
RC   EXCP   CONN    CPU    SRB  CLOCK   SERV  PG  PAGE  SWAP   VIO SWAPS 
00  11149  43739    .51    .01    1.8   389K   0 23536     0     0     0 
=15.51.44                                                                 
ATE          TOTAL TCB CPU TIME=   .51  TOTAL ELAPSED TIME=   1.8         

ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAM
ICE235I 0 OPTIONS: NULLOUT=RC0                                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN                                   
ICE750I 0 DC 3651621480 TC 0 CS DSVOO KSZ 21 VSZ 21                           
ICE752I 0 FSZ=5100030 RC  IGN=0 E  AVG=740 0  WSP=4901806 C  DYN=42415 53216   
ICE751I 1 DE-K61785 D5-K62201 D3-BASE   D7-BASE   E8-K61438                   
ICE090I 0 OUTPUT LRECL = 714, BLKSIZE = 27846, TYPE = FB   (SDB)               
ICE171I 0 SORTOUT LRECL OF 714 IS DIFFERENT FROM SORTIN(NN) LRECL OF 716 - RC=0
ICE055I 0 INSERT 0, DELETE 0                                                   
ICE054I 0 RECORDS - IN: 5100000, OUT: 5100000                                 
ICE134I 0 NUMBER OF BYTES SORTED: 3651600000                                   
ICE253I 0 RECORDS SORTED - PROCESSED: 5100000, EXPECTED: 5100030               
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 42450 , TRACKS USED: 32250     
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES                               
ICE180I 0 HIPERSPACE STORAGE USED = 2200084K BYTES                             
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES                                   
ICE052I 0 END OF DFSORT                                                       


No significant difference, back to the OP.


Gerry
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM

 

Similar Topics
Topic Forum Replies
No new posts Need to set RC4 through JCL SORT DFSORT/ICETOOL 5
No new posts How to split large record length file... DFSORT/ICETOOL 10
No new posts JCL sort card - get first day and las... JCL & VSAM 9
No new posts To get the the current time DFSORT/ICETOOL 13
No new posts RC query -Time column CA Products 3
Search our Forums:

Back to Top
IBMMainframes.com is not an official and/or affiliated with IBM® in anyway
Board Rules | FAQ | Downloads | Wiki | SiteMap | Contact Us