|
View previous topic :: View next topic
|
| Author |
Message |
rkarthik22
New User
Joined: 18 Apr 2009
Posts: 47
Location: India
|
|
|
|
Hai all,
In step 1, i am removing the header and trailer records from the input file having 132471138 records
| Quote: |
//STEP001 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTIN DD DISP=SHR,DSN=<input file> (having '132471138' number of records)
//SORTOUT DD DSN=<output file 1>, -------> input of step 2
// DISP=(NEW,CATLG,DELETE),
// UNIT=SYSDA,
// SPACE=(CYL,(900,800),RLSE),
// DATACLAS=DCCOM,
// VOL=(,,,80),
// DCB=*.STEP001.SORTIN
//SYSIN DD *
SORT FIELDS=COPY
OMIT COND=(01,1,CH,EQ,C'H',OR,
01,1,CH,EQ,C'T')
/*
|
In this step 2, i am just sorting and outrecing some necessary fields and also removing the duplicates
| Quote: |
//STEP002 EXEC PGM=SORT,
//SORTIN DD DSN=<output file 1>,
// DISP=SHR
//*
//SORTOUT DD DSN=<final output file>,
// DISP=(NEW,CATLG,DELETE),
// DCB=*.SORTIN,SPACE=&SPACE,
// DATACLAS=&DATACLAS,VOL=&VOL
//*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=(1,11,CH,A)
OUTREC FIELDS=(1:1,11,12:53,20,32:13,40,72:421,10,
82:275,09)
SUM FIELDS=NONE
/*
my question is, is there any other way to do this in sort methods...because its taking almost 3 hours to complete this job ... i know the input file size is of huge volume...but is there any way to reduce the time atleast...please help.. |
|
|
| Back to top |
|
 |
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix
|
|
|
|
Hello,
The volume is only a part of the problem. . .
What reason is there to copy the entire file to skip the "H" and "T" records?
How long are the records?
The jcl specifies the output use the same dcb as the input - is that correct? If only some of the data is needed, why?
Why have you posted in the DFSORT part of the forum, your system uses Syncsort. . . |
|
| Back to top |
|
|
rkarthik22
New User
Joined: 18 Apr 2009
Posts: 47
Location: India
|
|
|
|
hereafter, I will post the query in the correct forum !!! sorry
H - Header records is of 15 bytes starting from the first position
T - Trailer records id of 25 bytes starting from the first position..
input file records '132471138'
[quote]The jcl specifies the output use the same dcb as the input - is that correct? If only some of the data is needed, why?
| Quote: |
that is not a problem for me...i can use independent DISP & space properties for the step2....
the main thing ..is there anyway to reduce the run time ...thats it.. |
|
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix
|
|
|
|
Hello,
| Quote: |
| is there anyway to reduce the run time . |
Yes, surely. . . which is why i asked
| Quote: |
| What reason is there to copy the entire file to skip the "H" and "T" records? |
This is 100million+ unnecessary reads and writes, not counting an unnecessary very large transient file to pass from step to step.
Suggest you at least accomplish discarding the headers/trailers and sorting the needed data in one step.
You didn't answer how long the main records (the data records) are. If only part of the data is needed as output, it would be wise to only sort and write out the needed fields. I suspect that with a bit of re-work, the process might be reduced by well more than 1/2. |
|
| Back to top |
|
|
guptae
Moderator
Joined: 14 Oct 2005
Posts: 1209
Location: Bangalore,India
|
|
|
|
Hello Karthik,
Please try below sort card if
-- Put special key of '1' in 81 for data records.
INREC IFTHEN=(WHEN=INIT,OVERLAY=(81:C'1')),
-- For header record, put special key of '' in 81.
IFTHEN=(WHEN=(1,1,CH,EQ,C'H',OVERLAY=(81:C'')),
-- For trailer record, put special key of '9' in 81.
IFTHEN=(WHEN=(1,1,CH,EQ,C'T'),OVERLAY=(81:C'9'))
-- Sort by special key ('0', '1' or '9') and then regular key.
SORT FIELDS=(81,1,CH,A,14,10,CH,A)
--Remove special key.
OUTREC FIELDS=(1,80)
Here i assumed that record length is 80. please change it accordingly & see if it helps. |
|
| Back to top |
|
|
rkarthik22
New User
Joined: 18 Apr 2009
Posts: 47
Location: India
|
|
|
|
Hai dick,
the input file LRECL is 1602..
hai guptae!
| Quote: |
-- Put special key of '1' in 81 for data records.
INREC IFTHEN=(WHEN=INIT,OVERLAY=(81:C'1')),
-- For header record, put special key of '' in 81.
IFTHEN=(WHEN=(1,1,CH,EQ,C'H',OVERLAY=(81:C'')),
-- For trailer record, put special key of '9' in 81.
IFTHEN=(WHEN=(1,1,CH,EQ,C'T'),OVERLAY=(81:C'9'))
-- Sort by special key ('0', '1' or '9') and then regular key.
SORT FIELDS=(81,1,CH,A,14,10,CH,A)
--Remove special key.
OUTREC FIELDS=(1,80)
|
why should i use special key and all...i confused?! |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix
|
|
|
|
Hello,
The suggestion from Ekta is for cases when the header and trailer are to be preserved and the detail records sorted by some key and the final output has the original header, the newly sorted detail, and the original trailer.
In your first pass, it appeared that you intended to discard the header and trailer records. . . |
|
| Back to top |
|
|
Arun Raj
Moderator
Joined: 17 Oct 2006
Posts: 2482
Location: @my desk
|
|
|
|
rkarthik22,
Your input LRECL is as big as 1602 and you just need 90 bytes out of it in the ouput. Hence you must be using INREC instead of OUTREC as INREC selects the required fields before sort processing. And as Dick suggested, you dont really need two passes of data to achieve this.
Try this and see if the "run time" reduces.
| Code: |
//SYSIN DD *
OMIT COND=(1,1,SS,EQ,C'H,T')
INREC BUILD=(1,11,53,20,13,40,421,10,275,9)
SORT FIELDS=(1,11,CH,A) |
|
|
| Back to top |
|
|
rkarthik22
New User
Joined: 18 Apr 2009
Posts: 47
Location: India
|
|
|
|
Dick,
I dont want header and trailer in my final output file .
Thats why i omitted/removed the header and trailers which starts with H & T.
Hi arcvns
| Quote: |
//SYSIN DD *
OMIT COND=(1,1,SS,EQ,C'H,T')
INREC BUILD=(1,11,53,20,13,40,421,10,275,9)
SORT FIELDS=(1,11,CH,A) |
I will try this and let u know..thanks |
|
| Back to top |
|
|
rkarthik22
New User
Joined: 18 Apr 2009
Posts: 47
Location: India
|
|
|
|
Hi arcvns,
Thanks a lot!
| Quote: |
//SYSIN DD *
OMIT COND=(1,1,SS,EQ,C'H,T')
INREC BUILD=(1,11,53,20,13,40,421,10,275,9)
SORT FIELDS=(1,11,CH,A) |
It tooks only 26 minutes to run this job. And i got the expected output .(my earlier JCL tooks almost 3 hours to get the output)
thank you very much!!! |
|
| Back to top |
|
|
Arun Raj
Moderator
Joined: 17 Oct 2006
Posts: 2482
Location: @my desk
|
|
|
|
You're welcome. Glad that I could save you a couple of "hours"  |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix
|
|
|
|
Hello,
| Quote: |
| It tooks only 26 minutes to run this job. And i got the expected output .(my earlier JCL tooks almost 3 hours to get the output) |
Suggest you learn that it is better to think thru a process rather than just shoving together some jcl and control statements. . .
Multiple passes of large files is always questionable (in some cases it might not be wrong, but it is always worth looking into). If you find your design will do this, suggest looking for alternatives immediately. If you cannot find an alternative, ask for help.
If you are a programmer, think how you would accomplish the requirement if you wrote code. Then "program" the sort or other utility to behave similarly. |
|
| Back to top |
|
|
|
|
  |
All times are GMT + 6 Hours |
|
