Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

how to reduce the run time for the below sort

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM
View previous topic :: :: View next topic  
Author Message
maddukurivasu
Warnings : 1

New User


Joined: 14 Apr 2006
Posts: 4

PostPosted: Fri Feb 17, 2012 2:07 pm    Post subject: how to reduce the run time for the below sort
Reply with quote

My input file having 5 million records it takes 2 hours time to complete the job using below sort. please any one help me to reduce the time if any other option.

Code:
  SORT FIELDS=(19,15,ZD,A,715,2,ZD,A)
  OUTREC IFTHEN=(WHEN=INIT,
          BUILD=(1:1,714)),
       IFTHEN=(WHEN=(1,3,CH,EQ,C'HDR'),
          BUILD=(1:1,18,19:34,681,700:15X)),
       IFTHEN=(WHEN=(1,3,CH,EQ,C'TRL'),
          BUILD=(1:1,18,19:34,681,700:15X)),
       IFTHEN=(WHEN=(1,3,CH,NE,C'HDR',AND,1,3,CH,NE,C'TRL'),
        BUILD=(1:1,3,4:SEQNUM,15,ZD,START=1,19:19,696))
Back to top
View user's profile Send private message

Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7228

PostPosted: Fri Feb 17, 2012 2:15 pm    Post subject: Reply to: how to reduce the run time for the below sort
Reply with quote

Can you confirm which sort product you are using please?

Also LRECL and RECFM.

You are throwing around some large chunks of data twice per record. If you are able to describe your processing exactly, there may be some possibilities to reduce things, using OVERLAY, for instance.
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10202
Location: italy

PostPosted: Fri Feb 17, 2012 2:27 pm    Post subject: Reply to: how to reduce the run time for the below sort
Reply with quote

premise ... I an not a sort expert, but usually when You are able to get a result using <less> statements it might give some advantages
no reason to use more statements than needed
why not simplify the compare logic...
You are in an IF THEN / ELSE situation
if You check for HDR and TRL for equality, no reason to check also for inequality

try something along the lines of
Code:
  SORT FIELDS=(19,15,ZD,A,715,2,ZD,A)
  OUTREC IFTHEN=(WHEN=(1,3,CH,EQ,C'HDR',OR,1,3,CH,EQ,C'TRL'),
          BUILD=(1:1,18,19:34,681,700:15X)),
       IFTHEN=(WHEN=NONE,
        BUILD=(1:1,3,4:SEQNUM,15,ZD,START=1,19:19,696))


checked with a simpler record/key setup, but the results are just the same
Back to top
View user's profile Send private message
maddukurivasu
Warnings : 1

New User


Joined: 14 Apr 2006
Posts: 4

PostPosted: Fri Feb 17, 2012 2:27 pm    Post subject: Reply to: how to reduce the run time for the below sort
Reply with quote

Input file LRECL= 716, Output file LRECL=714,

RECFM= FB for both files,
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7228

PostPosted: Sat Feb 18, 2012 1:00 am    Post subject: Re: Reply to: how to reduce the run time for the below sort
Reply with quote

enrico-sorichetti wrote:
[...]
Code:
  SORT FIELDS=(19,15,ZD,A,715,2,ZD,A)
  OUTREC IFTHEN=(WHEN=(1,3,CH,EQ,C'HDR',OR,1,3,CH,EQ,C'TRL'),
          BUILD=(1:1,18,19:34,681,700:15X)),
       IFTHEN=(WHEN=NONE,
        BUILD=(1:1,3,4:SEQNUM,15,ZD,START=1,19:19,696))


checked with a simpler record/key setup, but the results are just the same


maddukurivasu, this looks like a good start from enrico. Any comment from you? Any answer to my earlier request?

The above still has 19:19,696 - a 696-byte move of data which is already there, as far as I can tell (and without any help from you). If you feel you have to do the build just to drop off the last two bytes of the record (don't know, because you haven't said), you could try this in place of the build.
Code:
  OVERLAY=(1:1,3,4:SEQNUM,15,ZD,START=1))


That hasn't dealt with the record-length. So start with

Code:
  OUTREC IFOUTLEN=714,


If that is good for you, you will have removed two 700 byte moves per 5,000,000 records. That should be noticeable.

If you feel it still "too slow" have a look at your datasets: blocking; buffers; file types. Also consult your sort manual (we are assuming Syncsort from where you have posted) and look for information on performance/tuning and see if that covers anything you are doing.
Back to top
View user's profile Send private message
don.leahy

Active Member


Joined: 06 Jul 2010
Posts: 641
Location: Whitby, ON, Canada

PostPosted: Sat Feb 18, 2012 1:34 am    Post subject:
Reply with quote

Are you sure that the OUTREC is the problem?

How long does the SORT step take to run if you take out the OUTREC entirely?
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Sat Feb 18, 2012 4:55 am    Post subject:
Reply with quote

Hi,

I generated 1700000 records with HDR in pos 1-3
1700000 records with TRL in pos 1-3
1700000 records without either HDR or TRL in pos 1-3.

A total of 510000 records.

I ran the code provided by the OP and here are the results
Code:
TOTAL TCB CPU TIME=   .46  TOTAL ELAPSED TIME=   2.9


ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAM
ICE235I 0 OPTIONS: NULLOUT=RC0                                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN                                   
ICE750I 0 DC 3651621480 TC 0 CS DSVOO KSZ 21 VSZ 21                           
ICE752I 0 FSZ=5100030 RC  IGN=0 E  AVG=740 0  WSP=4901806 C  DYN=46340 53216   
ICE751I 1 DE-K61785 D5-K62201 D3-BASE   D7-BASE   E8-K61438                   
ICE090I 0 OUTPUT LRECL = 714, BLKSIZE = 27846, TYPE = FB   (SDB)               
ICE171I 0 SORTOUT LRECL OF 714 IS DIFFERENT FROM SORTIN(NN) LRECL OF 716 - RC=0
ICE055I 0 INSERT 0, DELETE 0                                                   
ICE054I 0 RECORDS - IN: 5100000, OUT: 5100000                                 
ICE134I 0 NUMBER OF BYTES SORTED: 3651600000                                   
ICE253I 0 RECORDS SORTED - PROCESSED: 5100000, EXPECTED: 5100030               
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 46350 , TRACKS USED: 31080     
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES                               
ICE180I 0 HIPERSPACE STORAGE USED = 2231132K BYTES                             
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES                                   
ICE052I 0 END OF DFSORT                                                       



This is using DFSORT but can't imagine the difference would be so dramatic betwwen the 2 products.


Is the input on tape or disk and whatt is the blocksize for both input and output ?

Gerry
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7228

PostPosted: Sat Feb 18, 2012 5:29 am    Post subject: Reply to: how to reduce the run time for the below sort
Reply with quote

So, this is good. Even if we cut the OUTREC processing by 90%, we won't notice :-)

I suspect a COPY would give very similar results, as you've probably not generated any "random" values for the keys, but it certainly serves our purpose. The OUTREC is not the problem, good spot Don and Gerry.

We need the JCL and the output messages. Also interesting to know how that 15-digit key is made up. Is it several things? If so, how many, and how much do the values vary?
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Sat Feb 18, 2012 10:32 am    Post subject:
Reply with quote

Hi Bill,

actually I had generated a sequence number as such

Code:
  OPTION COPY                                                         
  OUTFIL REPEAT=1700000,OVERLAY=(19:SEQNUM,15,ZD,START=0,             
                715:SEQNUM,02,ZD,START=0)                             


This time I decided to vary the sort to
Code:
  SORT FIELDS=(30,4,ZD,D,22,4,ZD,A,26,4,ZD,D,19,3,ZD,A,715,2,ZD,A)
  OUTREC IFTHEN=(WHEN=INIT,                                       
          BUILD=(1:1,714)),                                       
       IFTHEN=(WHEN=(1,3,CH,EQ,C'HDR'),                           
          BUILD=(1:1,18,19:34,681,700:15X)),                       
       IFTHEN=(WHEN=(1,3,CH,EQ,C'TRL'),                           
          BUILD=(1:1,18,19:34,681,700:15X)),                       
       IFTHEN=(WHEN=(1,3,CH,NE,C'HDR',AND,1,3,CH,NE,C'TRL'),       
        BUILD=(1:1,3,4:SEQNUM,15,ZD,START=1,19:19,696))           



The result
Code:
 
RC   EXCP   CONN    CPU    SRB  CLOCK   SERV  PG  PAGE  SWAP   VIO SWAPS 
00  11149  43739    .51    .01    1.8   389K   0 23536     0     0     0 
=15.51.44                                                                 
ATE          TOTAL TCB CPU TIME=   .51  TOTAL ELAPSED TIME=   1.8         

ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAM
ICE235I 0 OPTIONS: NULLOUT=RC0                                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN                                   
ICE750I 0 DC 3651621480 TC 0 CS DSVOO KSZ 21 VSZ 21                           
ICE752I 0 FSZ=5100030 RC  IGN=0 E  AVG=740 0  WSP=4901806 C  DYN=42415 53216   
ICE751I 1 DE-K61785 D5-K62201 D3-BASE   D7-BASE   E8-K61438                   
ICE090I 0 OUTPUT LRECL = 714, BLKSIZE = 27846, TYPE = FB   (SDB)               
ICE171I 0 SORTOUT LRECL OF 714 IS DIFFERENT FROM SORTIN(NN) LRECL OF 716 - RC=0
ICE055I 0 INSERT 0, DELETE 0                                                   
ICE054I 0 RECORDS - IN: 5100000, OUT: 5100000                                 
ICE134I 0 NUMBER OF BYTES SORTED: 3651600000                                   
ICE253I 0 RECORDS SORTED - PROCESSED: 5100000, EXPECTED: 5100030               
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 42450 , TRACKS USED: 32250     
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES                               
ICE180I 0 HIPERSPACE STORAGE USED = 2200084K BYTES                             
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES                                   
ICE052I 0 END OF DFSORT                                                       


No significant difference, back to the OP.


Gerry
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
This topic is locked: you cannot edit posts or make replies. Get a job submitted itself every time... arunsoods JCL & VSAM 3 Tue Nov 22, 2016 4:05 pm
No new posts Limit duplicate records in the SORT pshongal SYNCSORT 6 Mon Nov 21, 2016 12:54 pm
No new posts How to convert the VBM file to VB or... Sulabh Agrawal JCL & VSAM 4 Fri Nov 18, 2016 1:04 pm
No new posts Check System time(Minute) Using TIME1... balaji81_k DFSORT/ICETOOL 5 Fri Nov 11, 2016 10:53 am
No new posts Sort records based on numeric field. Alks SYNCSORT 2 Wed Oct 19, 2016 10:14 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us