IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Sort step running long


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
Rijit

Active User


Joined: 15 Apr 2010
Posts: 168
Location: Pune

PostPosted: Wed Mar 21, 2012 1:04 pm
Reply with quote

I have a sort step which is running very long. Is there something we can do to tweak this one?

Elapsed time : 03.41.50.13
TCB time : 00.39.46.63

The input dataset is on DASD with record count 111442286. The sort step log information is posted below.

Code:

.   SORT FIELDS=COPY                                                           
.   OUTFIL FNAMES=SORTOUT1,INCLUDE=(1709,10,PD,NE,1000001)                     
.   OUTFIL FNAMES=SORTOUT2,INCLUDE=(1709,10,PD,EQ,1000001),                     
.   OUTREC=(1:1,1708)                                                           
.   OUTFIL FNAMES=SORTOUT3,INCLUDE=(1709,10,PD,EQ,1000001),                     
.   OUTREC=(1:1,1682)                                                           
. WER276B  SYSDIAG= 5344539, 9244769, 9244769, 5977050                         
. WER164B  10,924K BYTES OF VIRTUAL STORAGE AVAILABLE, MAX REQUESTED,           
. WER164B     0 BYTES RESERVE REQUESTED, 3,229,620 BYTES USED                   
. WER146B  20K BYTES OF EMERGENCY SPACE ALLOCATED                               
. WER108I  SORTIN   : RECFM=FB   ; LRECL=  1745; BLKSIZE= 27920                 
. WER110I  SORTOUT1 : RECFM=FB   ; LRECL=  1745; BLKSIZE= 31410                 
. WER110I  SORTOUT2 : RECFM=FB   ; LRECL=  1708; BLKSIZE= 32452                 
. WER110I  SORTOUT3 : RECFM=FB   ; LRECL=  1682; BLKSIZE= 31958                 
. WER410B  7,848K BYTES OF VIRTUAL STORAGE AVAILABLE ABOVE THE 16MEG LINE,     
. WER410B     0 BYTES RESERVE REQUESTED, 109,372 BYTES USED                     
. WER405I  SORTOUT1 :  DATA RECORDS OUT      78437; TOTAL RECORDS OUT      78437
. WER405I  SORTOUT2 :  DATA RECORDS OUT  111363849; TOTAL RECORDS OUT  111363849
. WER405I  SORTOUT3 :  DATA RECORDS OUT  111363849; TOTAL RECORDS OUT  111363849
 
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK

PostPosted: Wed Mar 21, 2012 1:08 pm
Reply with quote

As you are using SYNCSORT and this is a DFSORT part of the forum you will not get far. SYNCSORT questions get posted in the JCL part of the forum. No doubt some kind moderator will move it.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed Mar 21, 2012 1:09 pm
Reply with quote

If you contact Syncsort support, I'm sure they will be glad to help. If your search around in the JCL forum, not in the DFSORT forum, then you willl even find some appropriate contact e-mail addresses.

One thing I'd do, rather than wirting out two files that are pretty-much the same, but just a bit more data at the end of the record in one of them, is to just write out one file, and, if possible, change the programs reading the "new" file so they can read a bigger record but still do nothing with the extra bit of data.

EDIT: Sheesh, you are only copying. What do you do with the output files? Why is a COPY taking so much CPU? I suppose that is your question.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Wed Mar 21, 2012 1:51 pm
Reply with quote

unless these are tape,
am wondering about block size
Code:

WER108I  SORTIN   : RECFM=FB   ; LRECL=  1745; BLKSIZE= 27920                 
. WER110I  SORTOUT1 : RECFM=FB   ; LRECL=  1745; BLKSIZE= 31410                 
. WER110I  SORTOUT2 : RECFM=FB   ; LRECL=  1708; BLKSIZE= 32452                 
. WER110I  SORTOUT3 : RECFM=FB   ; LRECL=  1682; BLKSIZE= 31958   
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed Mar 21, 2012 2:00 pm
Reply with quote

Yes, seems odd to save 63 bytes of DASD per record, only to waste 24,000+ per block (if on DASD). The bigger rip-off on then would be reading later.

You are doing 222 million "moves" of around 1700 bytes, just to shorten the records. Has to be a better way, even if only by putting a dummy "IFTHEN" with IFOUTLEN= and the desired record length.

OK then, answer the obvious questions.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Wed Mar 21, 2012 2:51 pm
Reply with quote

IF DASD, the poor blocking would cause the excp count to be very high,
(1 block is a physical i/o)
So, i would hazard a guess that
(if DASD is the output
we have no idea, since pertinent info is missing:
  • DD statements from JCL
  • jesmsg info on i/o
  • WHAT ELSE IS GOING ON IN THE MACHINE WHILE THIS JOB RUNS
)
the job is waiting for i/o channel activity.

for someone with 133 posts, the TS has provided us with nothing except,
My job is running a long time
and posted in the wrong forum.
Back to top
View user's profile Send private message
Rijit

Active User


Joined: 15 Apr 2010
Posts: 168
Location: Pune

PostPosted: Wed Mar 21, 2012 2:57 pm
Reply with quote

dbzTHEdinosauer wrote:
unless these are tape,
am wondering about block size
Code:

WER108I  SORTIN   : RECFM=FB   ; LRECL=  1745; BLKSIZE= 27920                 
. WER110I  SORTOUT1 : RECFM=FB   ; LRECL=  1745; BLKSIZE= 31410                 
. WER110I  SORTOUT2 : RECFM=FB   ; LRECL=  1708; BLKSIZE= 32452                 
. WER110I  SORTOUT3 : RECFM=FB   ; LRECL=  1682; BLKSIZE= 31958   


I confirmed by checking the JCL all the 3 output datasets are on DASD not tapes. And they have DCB=(RECFM=FB,LRECL=1745,BLKSIZE=0) coded in so the BLKSIZE is calculated by the system.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8796
Location: Welsh Wales

PostPosted: Wed Mar 21, 2012 3:01 pm
Reply with quote

Stand well away from the doors please

Topic being moved to the JCL forum

Hold tight
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8796
Location: Welsh Wales

PostPosted: Wed Mar 21, 2012 3:03 pm
Reply with quote

Have you actually tried calculating the optimum DASD blocksize for each of these outputs and coding into the JCL and see if this helps.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Wed Mar 21, 2012 3:09 pm
Reply with quote

don"t know about syncsort,
but DFSORT,
you are best not to declare a record size for output file if you are using BUILD, etc....
Back to top
View user's profile Send private message
Rijit

Active User


Joined: 15 Apr 2010
Posts: 168
Location: Pune

PostPosted: Wed Mar 21, 2012 3:13 pm
Reply with quote

dbzTHEdinosauer wrote:
IF DASD, the poor blocking would cause the excp count to be very high,
(1 block is a physical i/o)
So, i would hazard a guess that
(if DASD is the output
we have no idea, since pertinent info is missing:
  • DD statements from JCL
  • jesmsg info on i/o
  • WHAT ELSE IS GOING ON IN THE MACHINE WHILE THIS JOB RUNS
)
the job is waiting for i/o channel activity.

for someone with 133 posts, the TS has provided us with nothing except,
My job is running a long time
and posted in the wrong forum.


Sorry for delayed response. Here is the expanded JCL for this step


Code:



XX         PARM='PGMN=SORT,ABEND=0004,DYNALLOC=(SYSDA,16),VSCORET=64M'
XXSYSPRINT DD  SYSOUT=*                                               
XXSYSOUT   DD  SYSOUT=*                                               
XXSYSUDUMP DD  SYSOUT=I                                               
&&SORTIN   DD  DSN=&SORTIN.,DISP=SHR                                 
XXSORTIN   DD  DSN=INVP.PRDM.INVOICE.MISC.UNLD.FULL,DISP=SHR   
&&SORTOUT1  DD  DSN=&SORTOUT1.,                                       
&&             DISP=(NEW,CATLG,DELETE),                               
&&             SPACE=&SPACE,DATACLAS=DCCOMP,                         
&&             DCB=(RECFM=FB,LRECL=1745,BLKSIZE=0)                   
XXSORTOUT1  DD  DSN=INVP.PRDM.INVOICE.MISC.UNLD.TX,               
XX             DISP=(NEW,CATLG,DELETE),                               
XX             SPACE=(CYL,(200,100),RLSE),DATACLAS=DCCOMP,           
XX             DCB=(RECFM=FB,LRECL=1745,BLKSIZE=0)                   
&&SORTOUT2  DD  DSN=&SORTOUT2.,                                       
&&             DISP=(NEW,CATLG,DELETE),                               
&&             SPACE=&SPACE,DATACLAS=DCCOMP,                         
&&             DCB=(RECFM=FB,LRECL=1708,BLKSIZE=0)                   
XXSORTOUT2  DD  DSN=INVP.PRDM.INVOICE.MISC.UNLD.NEW,
XX             DISP=(NEW,CATLG,DELETE),                       
XX             SPACE=(CYL,(200,100),RLSE),DATACLAS=DCCOMP,     
XX             DCB=(RECFM=FB,LRECL=1708,BLKSIZE=0)             
&&SORTOUT3  DD  DSN=&SORTOUT3.,                               
&&             DISP=(NEW,CATLG,DELETE),                       
&&             SPACE=&SPACE,DATACLAS=&DATACLAS,               
&&             DCB=(RECFM=FB,LRECL=1682,BLKSIZE=0)             
XXSORTOUT3  DD  DSN=INVP.PRDM.INVOICE.MISC.UNLD.IL,             
XX             DISP=(NEW,CATLG,DELETE),                       
XX             SPACE=(CYL,(200,100),RLSE),DATACLAS=DCCOMP,     
XX             DCB=(RECFM=FB,LRECL=1682,BLKSIZE=0)             
&&SYSIN    DD  DSN=&PARMLIB.(&MEM.),                           
&&             DISP=SHR                                       
XXSYSIN    DD  DSN=NBSP.WCC.PARMLIB(WC3SE63B),                 
XX             DISP=SHR                                       
  SORT FIELDS=COPY                                                           
.   OUTFIL FNAMES=SORTOUT1,INCLUDE=(1709,10,PD,NE,1000001)                     
.   OUTFIL FNAMES=SORTOUT2,INCLUDE=(1709,10,PD,EQ,1000001),                     
.   OUTREC=(1:1,1708)                                                           
.   OUTFIL FNAMES=SORTOUT3,INCLUDE=(1709,10,PD,EQ,1000001),                     
.   OUTREC=(1:1,1682)


JESYMSG

Code:

.IEF373I STEP/DB2ABEND/START 2012078.1109                                     
.IEF374I STEP/DB2ABEND/STOP  2012078.1109 CPU    0MIN 00.00SEC SRB    0MIN 00.0
.IEF236I ALLOC. FOR WRTGMR56 PS0100 JS0200                                     
.IGD103I SMS ALLOCATED TO DDNAME JOBLIB                                       
.IGD103I SMS ALLOCATED TO DDNAME                                               
.IGD103I SMS ALLOCATED TO DDNAME                                               
.IEF237I JES2 ALLOCATED TO SYSPRINT                                           
.IEF237I JES2 ALLOCATED TO SYSOUT                                             
.IEF237I JES2 ALLOCATED TO SYSUDUMP                                           
.IGD103I SMS ALLOCATED TO DDNAME SORTIN                                       
.IGD17070I DATA SET
INVP.PRDM.INVOICE.MISC.UNLD.TX                     
.ALLOCATED SUCCESSFULLY WITH 1 STRIPE(S).                                     
.IGD17160I DATA SET INVP.PRDM.INVOICE.MISC.UNLD.TX                       
.IS ELIGIBLE FOR COMPRESSION                                                   
.IGD101I SMS ALLOCATED TO DDNAME (SORTOUT1)                                   
.       
DSN (INVP.PRDM.INVOICE.MISC.UNLD.TX           )                   
.        STORCLAS (PRD1000S) MGMTCLAS (PRDLG) DATACLAS (DCCOM)                 
.        VOL SER NOS= 3SS#JB                                                   
.IGD17070I DATA SET INVP.PRDM.INVOICE.MISC.UNLD.NEW                       
.ALLOCATED SUCCESSFULLY WITH 1 STRIPE(S).   
.IGD17160I DATA SET INVP.PRDM.INVOICE.MISC.UNLD.NEW                         
.IS ELIGIBLE FOR COMPRESSION                                                   
.IGD101I SMS ALLOCATED TO DDNAME (SORTOUT2)                                     
.        DSN (INVP.PRDM.INVOICE.MISC.UNLD.NEW          )                     
.        STORCLAS (PRD1000S) MGMTCLAS (PRDLG) DATACLAS (DCCOM)                 
.        VOL SER NOS= 3SS#1R                                                   
.IGD17070I DATA SET INVP.PRDM.INVOICE.MISC.UNLD.IL                             
.ALLOCATED SUCCESSFULLY WITH 1 STRIPE(S).                                       
.IGD17160I DATA SET INVP.PRDM.INVOICE.MISC.UNLD.IL                               
.IS ELIGIBLE FOR COMPRESSION                                                   
.IGD101I SMS ALLOCATED TO DDNAME (SORTOUT3)                                     
.       
 DSN (INVP.PRDM.INVOICE.MISC.UNLD.IL               )                     
.        STORCLAS (PRD1000S) MGMTCLAS (PRDLG) DATACLAS (DCCOM)                 
.        VOL SER NOS= 3SSLQ2                                                   
.IGD103I SMS ALLOCATED TO DDNAME SYSIN                                         
.ACC20210-A ADDVOL FOR DD=SORTOUT2 DSN=INVP.PRDM.INVOICE.MISC.UNLD.NEW VOL=3SS
.ACC20600-A VOLUME * WAS ADDED TO DATA SET                                     
.ACC20210-A ADDVOL FOR DD=SORTOUT3 DSN=INVP.PRDM.INVOICE.MISC.UNLD.IL VOL=3SSLQ2
.ACC20600-A VOLUME * WAS ADDED TO DATA SET                                     
.IEF142I WRTGMR56 PS0100 JS0200 - STEP WAS EXECUTED - COND CODE 0000                                             
Back to top
View user's profile Send private message
Rijit

Active User


Joined: 15 Apr 2010
Posts: 168
Location: Pune

PostPosted: Wed Mar 21, 2012 3:22 pm
Reply with quote

Bill Woodger wrote:
If you contact Syncsort support, I'm sure they will be glad to help. If your search around in the JCL forum, not in the DFSORT forum, then you willl even find some appropriate contact e-mail addresses.

One thing I'd do, rather than wirting out two files that are pretty-much the same, but just a bit more data at the end of the record in one of them, is to just write out one file, and, if possible, change the programs reading the "new" file so they can read a bigger record but still do nothing with the extra bit of data.

EDIT: Sheesh, you are only copying. What do you do with the output files? Why is a COPY taking so much CPU? I suppose that is your question.


Hi Bill,

Very valid point. I am analysing on this. icon_smile.gif
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed Mar 21, 2012 3:28 pm
Reply with quote

Looking further lead to my previous question about what you are acutally doing with the files.

The little one it obviously makes sense to have seperate at some point. The bigger ones, less so (now that we know they aren't on tape, going to different locations, or similar).

How is the input being created? Does it come out of a SORT/TOOL step at any point? That could be a good time to create the small file.

The big output files are being processed at some point. How about, reading the big input file for your first application, doing logic to select and ignore the small sub-set to a new file, and just going with the big files for the rest of the processing, as mentioned above.

Then the whole kit-and-kaboodle just disappears.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed Mar 21, 2012 3:49 pm
Reply with quote

I meant "big file" singular in the above. Just use the main input file for the rest of the processing. You are "saving" little bits of DASD (subject to your blocking problem) yet having three copies of 111 million records of approx. 1700 bytes!

If you just use the main input file in place of the two you are creating, you save all that time/processing/cost and a whole heap of DASD, backup time/media, JCL simplicity, design simplicity, etc.

The best way to "save" CPU/IO is not usually by "tuning" as such, but by working out how things which are being done don't need to be done.

Pity I don't have some way to charge you for all those savings... :-)
Back to top
View user's profile Send private message
Rijit

Active User


Joined: 15 Apr 2010
Posts: 168
Location: Pune

PostPosted: Wed Mar 21, 2012 8:36 pm
Reply with quote

I checked the 3rd file with 1682 LRECL can be avoided. There is an easytrieve program which uses the 3rd file to get some data form it. So it can get the data from the bigger file also. Thanks a ton! icon_biggrin.gif
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed Mar 21, 2012 9:43 pm
Reply with quote

No problem. Now, If you can only use the big file instead of the 1708, and get the little file created somewhere else...
Back to top
View user's profile Send private message
Rijit

Active User


Joined: 15 Apr 2010
Posts: 168
Location: Pune

PostPosted: Thu Mar 22, 2012 12:38 am
Reply with quote

Bill Woodger wrote:
No problem. Now, If you can only use the big file instead of the 1708, and get the little file created somewhere else...


We need at least the first two files for our programs to work.

Anyways the little file has to be created in a downstream job as it is also required by another program. So it will consume the same CPU later in that step.

But I guess the advantage here is if we create the little file before then the SORTOUT2 will have less records to process by the EZT pgm in the same job It will have the records of the little file (SORTOUT1) eliminated in the sort step..Am I corect?
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Thu Mar 22, 2012 1:03 am
Reply with quote

The following is extracted from what you posted earlier.

Code:
.   SORT FIELDS=COPY                                                           
.   OUTFIL FNAMES=SORTOUT1,INCLUDE=(1709,10,PD,NE,1000001)                     
.   OUTFIL FNAMES=SORTOUT2,INCLUDE=(1709,10,PD,EQ,1000001),                     
.   OUTREC=(1:1,1708)                                                           
.   OUTFIL FNAMES=SORTOUT3,INCLUDE=(1709,10,PD,EQ,1000001),                     
.   OUTREC=(1:1,1682)                                                           
. WER108I  SORTIN   : RECFM=FB   ; LRECL=  1745; BLKSIZE= 27920                 
. WER110I  SORTOUT1 : RECFM=FB   ; LRECL=  1745; BLKSIZE= 31410                 
. WER110I  SORTOUT2 : RECFM=FB   ; LRECL=  1708; BLKSIZE= 32452                 
. WER110I  SORTOUT3 : RECFM=FB   ; LRECL=  1682; BLKSIZE= 31958                 
. WER405I  SORTOUT1 :  DATA RECORDS OUT      78437; TOTAL RECORDS OUT      78437
. WER405I  SORTOUT2 :  DATA RECORDS OUT  111363849; TOTAL RECORDS OUT  111363849
. WER405I  SORTOUT3 :  DATA RECORDS OUT  111363849; TOTAL RECORDS OUT  111363849
 


You have one input file with 111 million records on it.

You have three output files.

The first output file contains 78,437 records. This is less than 0.008% of the file. The Easytrieve will hardly notice if it has to read those extra records and ignore them.

The two large output files are almost the same as the large input file, just missing some little bits at the back, 37 bytes for the first and 55 for the second. If you change whatever uses those files to work with the little extra length, then you save yourself writing and storing those two files.

Now, you don't have to specifically read the main input to create the small file. In the first place where you have to read a big file, you can amend to also create the small file. This will add very little processing time to that program and hardly overburden it with complexity.

So, if you run everything off your existing main input file, creating the small file the first time the main input file is read by a program (or at least some time before you need to read the small file) then:


  • You change all programs to operate with the size of the existing main file
  • You need new code to create the small file from somewhere where the main input file is already being read
  • You need to code to ignore the records from the small file as they now remain on the big file


In exchange for the above work, you completely remove the job which is running slowly. The additional resources necessary on a daily basis is just the minimal stuff to identify and write the records to the small file from an application program of your choice which is already reading the main input file.

To get these benfits, you have minor coding changes to application programs.

These benefits are, in a year, saving about a month of elapsed time and five-and-a-half days of CPU time. Plus you save on holding two additional copies of the vast majority of your main input file on DASD. I just tried to calculate how much that would be in Cylinders and broke my calculator.

If there is any sort of genuine business, or technical, reason why that can't be done for a file containing 111 million 1700-byte records, I'd like to know it. The "designer" who came up with this in the first place should be encouraged to wear an absurd paper-hat at work for at least a month.

Just change all the programs to read the main file, ignoring any data they don't want. Ditch the SORT/COPY. Create the small file the first time a big file is read. OK, if that can't be done and keep your programs "working", what more can I say? :-)
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM

 


Similar Topics
Topic Forum Replies
No new posts JCL sort to compare dates in two file... DFSORT/ICETOOL 2
No new posts Is this possible via sort (in one pass)? SYNCSORT 4
No new posts GDG generation name to GDG Base name ... DFSORT/ICETOOL 3
No new posts SORT on detail record, then repeat he... DFSORT/ICETOOL 3
No new posts FB to .CSV conversion using sort DFSORT/ICETOOL 7
Search our Forums:

Back to Top