Portal | Manuals | References | Downloads | Info | Programs | JCLs | Mainframe wiki | Quick Ref
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Profile Log in to check your private messages Log in
 
Reduce the run time in SORT

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM
View previous topic :: :: View next topic  
Author Message
rkarthik22

New User


Joined: 18 Apr 2009
Posts: 47
Location: India

PostPosted: Mon May 25, 2009 8:43 am    Post subject: Reduce the run time in SORT
Reply with quote

Hai all,

In step 1, i am removing the header and trailer records from the input file having 132471138 records

Quote:
//STEP001 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTIN DD DISP=SHR,DSN=<input file> (having '132471138' number of records)
//SORTOUT DD DSN=<output file 1>, -------> input of step 2
// DISP=(NEW,CATLG,DELETE),
// UNIT=SYSDA,
// SPACE=(CYL,(900,800),RLSE),
// DATACLAS=DCCOM,
// VOL=(,,,80),
// DCB=*.STEP001.SORTIN
//SYSIN DD *
SORT FIELDS=COPY
OMIT COND=(01,1,CH,EQ,C'H',OR,
01,1,CH,EQ,C'T')
/*




In this step 2, i am just sorting and outrecing some necessary fields and also removing the duplicates


Quote:
//STEP002 EXEC PGM=SORT,
//SORTIN DD DSN=<output file 1>,
// DISP=SHR
//*
//SORTOUT DD DSN=<final output file>,
// DISP=(NEW,CATLG,DELETE),
// DCB=*.SORTIN,SPACE=&SPACE,
// DATACLAS=&DATACLAS,VOL=&VOL
//*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=(1,11,CH,A)
OUTREC FIELDS=(1:1,11,12:53,20,32:13,40,72:421,10,
82:275,09)
SUM FIELDS=NONE
/*


my question is, is there any other way to do this in sort methods...because its taking almost 3 hours to complete this job ... i know the input file size is of huge volume...but is there any way to reduce the time atleast...please help..
Back to top
View user's profile Send private message

dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Mon May 25, 2009 9:12 am    Post subject:
Reply with quote

Hello,

The volume is only a part of the problem. . .

What reason is there to copy the entire file to skip the "H" and "T" records?

How long are the records?

The jcl specifies the output use the same dcb as the input - is that correct? If only some of the data is needed, why?

Why have you posted in the DFSORT part of the forum, your system uses Syncsort. . .
Back to top
View user's profile Send private message
rkarthik22

New User


Joined: 18 Apr 2009
Posts: 47
Location: India

PostPosted: Mon May 25, 2009 9:51 am    Post subject: Re: Reduce the run time in SORT
Reply with quote

hereafter, I will post the query in the correct forum !!! sorry

H - Header records is of 15 bytes starting from the first position
T - Trailer records id of 25 bytes starting from the first position..

input file records '132471138'

[quote]The jcl specifies the output use the same dcb as the input - is that correct? If only some of the data is needed, why?
Quote:



that is not a problem for me...i can use independent DISP & space properties for the step2....

the main thing ..is there anyway to reduce the run time ...thats it..
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Mon May 25, 2009 10:03 am    Post subject:
Reply with quote

Hello,

Quote:
is there anyway to reduce the run time .
Yes, surely. . . which is why i asked
Quote:
What reason is there to copy the entire file to skip the "H" and "T" records?
This is 100million+ unnecessary reads and writes, not counting an unnecessary very large transient file to pass from step to step.

Suggest you at least accomplish discarding the headers/trailers and sorting the needed data in one step.

You didn't answer how long the main records (the data records) are. If only part of the data is needed as output, it would be wise to only sort and write out the needed fields. I suspect that with a bit of re-work, the process might be reduced by well more than 1/2.
Back to top
View user's profile Send private message
guptae

Moderator


Joined: 14 Oct 2005
Posts: 1187
Location: Bangalore,India

PostPosted: Mon May 25, 2009 10:16 am    Post subject: Reply to: Reduce the run time in SORT
Reply with quote

Hello Karthik,

Please try below sort card if

-- Put special key of '1' in 81 for data records.
INREC IFTHEN=(WHEN=INIT,OVERLAY=(81:C'1')),
-- For header record, put special key of '' in 81.
IFTHEN=(WHEN=(1,1,CH,EQ,C'H',OVERLAY=(81:C'')),
-- For trailer record, put special key of '9' in 81.
IFTHEN=(WHEN=(1,1,CH,EQ,C'T'),OVERLAY=(81:C'9'))
-- Sort by special key ('0', '1' or '9') and then regular key.
SORT FIELDS=(81,1,CH,A,14,10,CH,A)
--Remove special key.
OUTREC FIELDS=(1,80)

Here i assumed that record length is 80. please change it accordingly & see if it helps.
Back to top
View user's profile Send private message
rkarthik22

New User


Joined: 18 Apr 2009
Posts: 47
Location: India

PostPosted: Mon May 25, 2009 11:52 am    Post subject:
Reply with quote

Hai dick,

the input file LRECL is 1602..

hai guptae!

Quote:
-- Put special key of '1' in 81 for data records.
INREC IFTHEN=(WHEN=INIT,OVERLAY=(81:C'1')),
-- For header record, put special key of '' in 81.
IFTHEN=(WHEN=(1,1,CH,EQ,C'H',OVERLAY=(81:C'')),
-- For trailer record, put special key of '9' in 81.
IFTHEN=(WHEN=(1,1,CH,EQ,C'T'),OVERLAY=(81:C'9'))
-- Sort by special key ('0', '1' or '9') and then regular key.
SORT FIELDS=(81,1,CH,A,14,10,CH,A)
--Remove special key.
OUTREC FIELDS=(1,80)


why should i use special key and all...i confused?!
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Mon May 25, 2009 12:11 pm    Post subject:
Reply with quote

Hello,

The suggestion from Ekta is for cases when the header and trailer are to be preserved and the detail records sorted by some key and the final output has the original header, the newly sorted detail, and the original trailer.

In your first pass, it appeared that you intended to discard the header and trailer records. . .
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2285
Location: @my desk

PostPosted: Mon May 25, 2009 2:07 pm    Post subject:
Reply with quote

rkarthik22,

Your input LRECL is as big as 1602 and you just need 90 bytes out of it in the ouput. Hence you must be using INREC instead of OUTREC as INREC selects the required fields before sort processing. And as Dick suggested, you dont really need two passes of data to achieve this.

Try this and see if the "run time" reduces.
Code:
//SYSIN   DD *
  OMIT COND=(1,1,SS,EQ,C'H,T')           
  INREC BUILD=(1,11,53,20,13,40,421,10,275,9)
  SORT FIELDS=(1,11,CH,A)
Back to top
View user's profile Send private message
rkarthik22

New User


Joined: 18 Apr 2009
Posts: 47
Location: India

PostPosted: Mon May 25, 2009 4:29 pm    Post subject: Re: Reduce the run time in SORT
Reply with quote

Dick,

I dont want header and trailer in my final output file .

Thats why i omitted/removed the header and trailers which starts with H & T.

Hi arcvns

Quote:
//SYSIN DD *
OMIT COND=(1,1,SS,EQ,C'H,T')
INREC BUILD=(1,11,53,20,13,40,421,10,275,9)
SORT FIELDS=(1,11,CH,A)


I will try this and let u know..thanks
Back to top
View user's profile Send private message
rkarthik22

New User


Joined: 18 Apr 2009
Posts: 47
Location: India

PostPosted: Mon May 25, 2009 6:10 pm    Post subject:
Reply with quote

Hi arcvns,

Thanks a lot!

Quote:
//SYSIN DD *
OMIT COND=(1,1,SS,EQ,C'H,T')
INREC BUILD=(1,11,53,20,13,40,421,10,275,9)
SORT FIELDS=(1,11,CH,A)


It tooks only 26 minutes to run this job. And i got the expected output .(my earlier JCL tooks almost 3 hours to get the output)

thank you very much!!!
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2285
Location: @my desk

PostPosted: Mon May 25, 2009 6:34 pm    Post subject:
Reply with quote

You're welcome. Glad that I could save you a couple of "hours" icon_smile.gif
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Mon May 25, 2009 10:58 pm    Post subject:
Reply with quote

Hello,

Quote:
It tooks only 26 minutes to run this job. And i got the expected output .(my earlier JCL tooks almost 3 hours to get the output)
Suggest you learn that it is better to think thru a process rather than just shoving together some jcl and control statements. . .

Multiple passes of large files is always questionable (in some cases it might not be wrong, but it is always worth looking into). If you find your design will do this, suggest looking for alternatives immediately. If you cannot find an alternative, ask for help.

If you are a programmer, think how you would accomplish the requirement if you wrote code. Then "program" the sort or other utility to behave similarly.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Comparing 2 Files using Current time arunsoods DFSORT/ICETOOL 5 Fri Sep 22, 2017 6:00 pm
No new posts Difference in SORT & ICETOOL Mohan Kothakota DFSORT/ICETOOL 5 Fri Sep 22, 2017 4:56 pm
No new posts OC4 in SORT Mohan Kothakota DFSORT/ICETOOL 2 Tue Sep 12, 2017 6:30 pm
No new posts Regarding time parameter shanthi gude JCL & VSAM 7 Mon Sep 04, 2017 2:31 pm
No new posts SORT Trailer Count - LRECL Output co... amorante DFSORT/ICETOOL 5 Tue Aug 29, 2017 8:57 pm

Facebook
Back to Top
 
Job Vacancies | Forum Rules | Bookmarks | Subscriptions | FAQ | Polls | Contact Us