|
View previous topic :: View next topic
|
| Author |
Message |
shashikant
New User
.jpg)
Joined: 07 Sep 2007
Posts: 6
Location: chennai
|
|
|
|
Hi All,
I have one jobstep which merges 10 datasets.
Each of these datasets are having minium of one million records and this step is taking most of the time in merging these dataset itself,
can any one please help me in giving the best solution so that i can reduce the time taken for this step
Thanks,
Shashikanth |
|
| Back to top |
|
 |
expat
Global Moderator
Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales
|
|
|
|
| How do you do it now ? |
|
| Back to top |
|
|
shashikant
New User
Joined: 07 Sep 2007
Posts: 6
Location: chennai
|
|
|
|
This is plain concatenation of 10 datasets with in the same jobstep.
//skrao1 dd=dsn1,disp=shr
// dd=dsn2,disp=shr
// dd=dsn3,disp=shr
// dd=dsn4,disp=shr
// dd=dsn5,disp=shr
// dd=dsn6,disp=shr
// dd=dsn7,disp=shr
// dd=dsn8,disp=shr
// dd=dsn9,disp=shr
// dd=dsn10,disp=shr
Thanks,
Shashikanth |
|
| Back to top |
|
|
stodolas
Active Member
Joined: 13 Jun 2007
Posts: 632
Location: Wisconsin
|
|
|
|
| What is "most of the time" 20 minutes, 10 minutes? Wall time, cpu time? If a job runs for 8 hours and only uses 1 hour of CPU time that could be a good thing. If this step is taking 10 minutes that might not be a bad thing |
|
| Back to top |
|
|
expat
Global Moderator
Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales
|
|
|
|
| What program is being used to concatenate the data ? |
|
| Back to top |
|
|
Bill Dennis
Active Member
Joined: 17 Aug 2007
Posts: 562
Location: Iowa, USA
|
|
|
|
| Are the datasets on tape or DASD? If tape, are they multi-volume? What record size/Block size? |
|
| Back to top |
|
|
Craq Giegerich
Senior Member
Joined: 19 May 2007
Posts: 1512
Location: Virginia, USA
|
|
|
|
| It doesn't even appear to be a merge! Some more detail would help us to help you! |
|
| Back to top |
|
|
Alissa Margulies
SYNCSORT Support
Joined: 25 Jul 2007
Posts: 496
Location: USA
|
|
|
|
Shashikanth,
I made the following assumptions:
1. All the datasets are FB and have the same LRECL.
2. All the data is already in sorted order.
3. The SORT control field starts in position 1 for the length of 10.
Try the following (modify based on your actual data):
| Code: |
//STEP1 EXEC PGM=SORT
//SORTIN01 DD DSN=FILE1,DISP=SHR
//SORTIN02 DD DSN=FILE2,DISP=SHR
//SORTIN03 DD DSN=FILE3,DISP=SHR
//SORTIN04 DD DSN=FILE4,DISP=SHR
//SORTIN05 DD DSN=FILE5,DISP=SHR
//SORTIN06 DD DSN=FILE6,DISP=SHR
//SORTIN07 DD DSN=FILE7,DISP=SHR
//SORTIN08 DD DSN=FILE8,DISP=SHR
//SORTIN09 DD DSN=FILE9,DISP=SHR
//SORTIN10 DD DSN=FILE10,DISP=SHR
//SORTOUT DD DSN=MERGED.OUTPUT,DISP=...
//SYSOUT DD SYSOUT=*
//SYSIN DD *
MERGE FIELDS=(1,10,CH,A)
/* |
|
|
| Back to top |
|
|
UmeySan
Active Member
Joined: 22 Aug 2006
Posts: 771
Location: Germany
|
|
|
|
Hi shashikant !
Assuming same as Alissa Margulies, and that every DSN is having minium of one million records i would alloc more space to Sort by using the $ORTPARM Parameter.
Something like this:
VSCORE=5M
VSCORET=30M
Also try using BUFNO Parameter in DD for the Sortin-Files.
I also agree with stodolas !!!
Regards, UmeySan |
|
| Back to top |
|
|
ramangouda patil
New User
Joined: 09 Apr 2008
Posts: 39
Location: India
|
|
|
|
| Please correct me if I am wrong. Alissa, wouldn't we need to specify work files for the merging done using the SORT utility ? |
|
| Back to top |
|
|
Alissa Margulies
SYNCSORT Support
Joined: 25 Jul 2007
Posts: 496
Location: USA
|
|
|
|
Shashikanth,
If you have been testing the various suggestions made to you, but are still not seeing the desired performance improvements, then add the following to your sort step:
| Code: |
//$ORTPARM DD *
MSG=AP,LIST,GMSG |
This will provide additional tuning messages that will help me to better assist you. You can email me the complete job listing offline at alissa.margulies@syncsort.com.
(And as both UmeySan and stodolas pointed out, you still need to identify which you are looking to improve - Wall Clock or CPU time). |
|
| Back to top |
|
|
Alissa Margulies
SYNCSORT Support
Joined: 25 Jul 2007
Posts: 496
Location: USA
|
|
|
|
| ramangouda patil wrote: |
| Please correct me if I am wrong. Alissa, wouldn't we need to specify work files for the merging done using the SORT utility ? |
Not if dynamic allocation is enabled (either by the DYNALOC installation option or the DYNALLOC runtime parameter). |
|
| Back to top |
|
|
Alissa Margulies
SYNCSORT Support
Joined: 25 Jul 2007
Posts: 496
Location: USA
|
|
|
|
| UmeySan wrote: |
| Also try using BUFNO Parameter in DD for the Sortin-Files. |
FYI, BUFNO values are ignored by SyncSort under normal circumstances where SyncSort is handling the I/O. |
|
| Back to top |
|
|
ramangouda patil
New User
Joined: 09 Apr 2008
Posts: 39
Location: India
|
|
|
|
Alissa
| Code: |
//$ORTPARM DD *
MSG=AP,LIST,GMSG |
Could you elaborate a little on what this piece of code signifies ? |
|
| Back to top |
|
|
Alissa Margulies
SYNCSORT Support
Joined: 25 Jul 2007
Posts: 496
Location: USA
|
|
|
|
| ramangouda patil wrote: |
| Code: |
//$ORTPARM DD *
MSG=AP,LIST,GMSG |
Could you elaborate a little on what this piece of code signifies ? |
The $ORTPARM DD statement is used to override PARM or control statement information. In this case, I want to ensure that all the sort related message are written to the message dataset (MSG=AP), that the header lines and user control statements are also listed with the SYSOUT data set (LIST), and that additional tuning messages get posted (GMSG).
Hope this helps. |
|
| Back to top |
|
|
ramangouda patil
New User
Joined: 09 Apr 2008
Posts: 39
Location: India
|
|
|
|
One last doubt Alissa 
How does one look into the message dataset (MSG=AP) and GMSG (What is the path or where are they stored) ?
And how can one fine tune the sort looking into them (you can help a lot by taking examples say values for CU time and explain thus)
Thanks a lot! |
|
| Back to top |
|
|
Alissa Margulies
SYNCSORT Support
Joined: 25 Jul 2007
Posts: 496
Location: USA
|
|
|
|
| ramangouda patil wrote: |
| How does one look into the message dataset (MSG=AP) and GMSG (What is the path or where are they stored) ? |
Each site controls where they route the SyncSort messages to via the MSGDD installation option. The default name is SYSOUT. However, if you need to verify your current settings, you can run SYNCLIST which produces a report of all your current SyncSort installation options.
| ramangouda patil wrote: |
| And how can one fine tune the sort looking into them |
'G' messages contain internal information that are specifically designed for SyncSort technicians to analyze. |
|
| Back to top |
|
|
gcicchet
Senior Member
Joined: 28 Jul 2006
Posts: 1702
Location: Australia
|
|
|
|
Hi,
| Quote: |
| Please correct me if I am wrong. Alissa, wouldn't we need to specify work files for the merging done using the SORT utility ? |
Why do we need work files when MERGE is used, shouldn't the files be already sorted ?
Gerry |
|
| Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA
|
|
|
|
Alissa said:
| Quote: |
| Please correct me if I am wrong. Alissa, wouldn't we need to specify work files for the merging done using the SORT utility ? |
| Quote: |
| Not if dynamic allocation is enabled (either by the DYNALOC installation option or the DYNALLOC runtime parameter). |
Alissa,
Are you saying that Syncsort actually uses dynamic allocation for a MERGE? DFSORT doesn't so I'm surprised that Syncsort does.
I see people using //SORTWKdd DD files for COPY and MERGE quite often and I suggest they remove them to avoid wasting space, so I'd like to clear up the confusion here. |
|
| Back to top |
|
|
Arun Raj
Moderator
Joined: 17 Oct 2006
Posts: 2481
Location: @my desk
|
|
|
|
| Quote: |
| I see people using //SORTWKdd DD files for COPY and MERGE quite often and I suggest they remove them to avoid wasting space |
Frank,
Are you saying that SORTWKdd is not required for COPY and MERGE?
Thanks,
Arun |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
Yes, they are typically not needed for COPY and MERGE.
As the name suggests (SORTWKxx), they are typically work space used for the sort to massage the data to get from "un-sequenced" to the desired sequence.
COPY coes not care about a sequence and MERGE implies that the files to be merged are already in the required sequence. |
|
| Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA
|
|
|
|
| Quote: |
Frank,
Are you saying that SORTWKdd is not required for COPY and MERGE? |
Right - DFSORT does NOT use work data sets for COPY or MERGE, only for SORT. If you specify SORTWKdd DDs for COPY or MERGE, the system allocates the space, but DFSORT doesn't use the SORTWKdd datas sets, so the space is wasted. If you use dynamic allocation of work data sets instead of JCL SORTWKdd DDs (as recommended for DFSORT), DFSORT does not allocate any work space for COPY or MERGE, so space is not wasted. |
|
| Back to top |
|
|
Anuj Dhawan
Superior Member
Joined: 22 Apr 2006
Posts: 6250
Location: Mumbai, India
|
|
|
|
Hi,
Even SyncSort don't use any SORTWKdd files in MERGE, one of the LPARs at my shop uses SyncSort, used below JOB there for MERGE without SORTWKdd, it went fine..
| Code: |
//SORTIN1 DD DSN=HLQ.FIRST.FILE,
// DISP=SHR
//SORTIN2 DD DSN=HLQ.SECOND.FILE,
// DISP=SHR
//SORTOUT DD DSN=HLQ.MERGE.FILE,
// DISP=(NEW,CATLG,CATLG),
// DSORG=PS,
// RECFM=FB,
// LRECL=80,
// BLKSIZE=0,
// DATACLAS=LARGE,
// UNIT=SYSDA
//*SORTWK01 DD UNIT=SYSDA,SPACE=(CYL,(20,5))
//SYSIN DD *
MERGE FIELDS=(1,10,CH,A)
/* |
Mind you DD SORTWK01 is commented out.
I think, this conversation
| Quote: |
Please correct me if I am wrong. Alissa, wouldn't we need to specify work files for the merging done using the SORT utility ?
| Quote: |
| Not if dynamic allocation is enabled (either by the DYNALOC installation option or the DYNALLOC runtime parameter). |
|
might be because of fast fingers.
Hope this helps. |
|
| Back to top |
|
|
gcicchet
Senior Member
Joined: 28 Jul 2006
Posts: 1702
Location: Australia
|
|
|
|
Hi Anuj,
your example doesn't conclusively indicate whether or not WORK files are used as DYNALOC could be enabled hence you don't need to specify SORTWK files.
I think we need Alissa to answer that or maybe the job output may show information on whether SORTWK files are used.
I'd be surprised if they are needed.
Gerry |
|
| Back to top |
|
|
CICS Guy
Senior Member
Joined: 18 Jul 2007
Posts: 2146
Location: At my coffee table
|
|
|
|
| gcicchet wrote: |
| your example doesn't conclusively indicate whether or not WORK files are used as DYNALOC could be enabled hence you don't need to specify SORTWK files. |
Not to be dis-respective, but COPY never needs works, in ANY sort.
Merge requirements are that all inputs be in the same sequence, merge would fail if there was an 'out of sequence' condition.
Trust this, MERGE and COPY do NOT need work space! |
|
| Back to top |
|
|
|
|
