View previous topic :: View next topic
|
Author |
Message |
fazilbe
New User
Joined: 03 Dec 2008 Posts: 25 Location: INDIA
|
|
|
|
Hi,
I have a job which runs for close to 10 hours each month. When analyzed found that one particular PROC step consuming all the run time. The step just does a copy of a huge file to a GDG using IEBGENER. Input file has a more than 100gb of record. Can we make this step more faster using any other technique. Will SORT be faster compared to IEBGENER. Please help me solve this issue. Please find the JCL below.
Code: |
//JS010 EXEC AIP04000,
// RETPD='720',
// UNIT=CART,
// UNIT1='SYSDA',
// SPACEMY='100,50',
//PS040 EXEC PGM=IEBGENER,
// COND=(0,LT)
//SYSUT1 DD DSN=temp1,
// DISP=SHR
//SYSUT2 DD DSN=temp2(+1),
// VOL=(,,,99),
// FREE=CLOSE,
// DISP=(NEW,CATLG,DELETE),
// UNIT=&UNIT,
// LABEL=RETPD=&RETPD,
// DCB=(&DSCB)
//SYSIN DD DUMMY
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=Y
|
Regards,
Fazil |
|
Back to top |
|
|
Robert Sample
Global Moderator
Joined: 06 Jun 2008 Posts: 8697 Location: Dubuque, Iowa, USA
|
|
|
|
Add buffers to input and output file; with large files you should never use the z/OS default. |
|
Back to top |
|
|
expat
Global Moderator
Joined: 14 Mar 2007 Posts: 8797 Location: Welsh Wales
|
|
|
|
Quote: |
Will SORT be faster compared to IEBGENER. Please help me solve this issue. Please find the JCL below. |
May I suggest that you try it for yourself. The workload of your machine can not be replicated on ours, and this is one of the major factors involved.
What are the specified paramters for &DSCB and to they allow for SDB to be used. |
|
Back to top |
|
|
fazilbe
New User
Joined: 03 Dec 2008 Posts: 25 Location: INDIA
|
|
|
|
Hi,
DCB=(796 27,860 0 FB) is what i could find in the spool log. Does this help?. Please let me know if more information is needed.
Regards,
Fazil |
|
Back to top |
|
|
expat
Global Moderator
Joined: 14 Mar 2007 Posts: 8797 Location: Welsh Wales
|
|
|
|
I would replace that line with
Code: |
// RECFM=FB,LRECL=796 |
I assume that the output is tape and the code above will allow the system to determine the best output blocksize. This would allow more records per block to be written so less writes are required.
Is the input file tape also ? If so, is the blocksize optimum for tape or also defined for DASD ?
Also look at the suggestion by Robert to increase the I/O buffering, say 20 buffers per DD may significantly improve performance. |
|
Back to top |
|
|
Bill Dennis
Active Member
Joined: 17 Aug 2007 Posts: 562 Location: Iowa, USA
|
|
|
|
Another option would be to allocate multiple tape drives on input and output as
This is less effective on newer tape technology where the "end" is back at the "start" but if your tapes are virtual and are being recalled it might help. |
|
Back to top |
|
|
Douglas Wilder
Active User
Joined: 28 Nov 2006 Posts: 305 Location: Deerfield IL
|
|
|
|
Check how long it was between the time the system requested the tape to be mounted and the time the operator informed the system that the tape was mounted. If the operators do not mount the tapes in a timely manor no system changes can improve the time that much.
Do allow the system to choose the best block size and increate the buffers as stated by others, and change IEBGENER to ICEGENER (If it has been installed this way on your system). This will use sort to copy the file and my experience is that this will help quite a bit. |
|
Back to top |
|
|
Alissa Margulies
SYNCSORT Support
Joined: 25 Jul 2007 Posts: 496 Location: USA
|
|
|
|
If you are running SyncSort for z/OS, verify that the BetterGener function has been installed. The BetterGener utility program offers improved performance over IEBGENER in straight copy tasks and is transparent to users. Implementing BetterGener will provide significant savings in system resources such as elapsed time, CPU time, and channel utilization through improved I/O handling. Once BetterGener is activated, eligible IEBGENER job streams are automatically processed by SyncSort. There is no need to replace working IEBGENER job streams with explicit requests for SyncSort's more efficient copying. |
|
Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
|
|
|
|
Quote: |
Do allow the system to choose the best block size and increate the buffers as stated by others, and change IEBGENER to ICEGENER (If it has been installed this way on your system). This will use sort to copy the file and my experience is that this will help quite a bit. |
If DFSORT's ICEGENER has been installed as a replacement for IEBGENER, then you don't need to change PGM=IEBGENER to PGM=ICEGENER. ICEGENER will be used automatically. If DFSORT's ICEGENER has not been installed as a replacement for IEBGENER, then using PGM=ICEGENER would be necessary.
You don't need to increase the buffers for ICEGENER - it does its own optimal buffering.
For more information on DFSORT's ICEGENER, see:
publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/ICE1CA30/9.2.16?DT=20080528171007 |
|
Back to top |
|
|
gcicchet
Senior Member
Joined: 28 Jul 2006 Posts: 1702 Location: Australia
|
|
|
|
Hi,
I would also check the BLOCKSIZE of the input file.
Gerry |
|
Back to top |
|
|
fazilbe
New User
Joined: 03 Dec 2008 Posts: 25 Location: INDIA
|
|
|
|
Hi,
I will try the valuable suggestions given by everybody and let you all know the outcome of it. Thanks a lot.
Regards,
Fazil |
|
Back to top |
|
|
fazilbe
New User
Joined: 03 Dec 2008 Posts: 25 Location: INDIA
|
|
|
|
Hi,
Actually input is in DASD and the output GDG has been written to TAPE. |
|
Back to top |
|
|
skkp2006
New User
Joined: 14 Jul 2006 Posts: 93 Location: Chennai,India
|
|
|
|
Alissa,
Quote: |
"If you are running SyncSort for z/OS, verify that the BetterGener function has been installed. " |
How can we possibly check whether BetterGener has been installed in our shop? Does it come along with SyncSort or is it a pay and use product?
Syam |
|
Back to top |
|
|
Arun Raj
Moderator
Joined: 17 Oct 2006 Posts: 2481 Location: @my desk
|
|
|
|
Quote: |
Once BetterGener is activated, eligible IEBGENER job streams are automatically processed by SyncSort |
skkp2006,
As pointed out by Alissa, you can run a small IEBGENER copy and check if SyncSort gets invoked. |
|
Back to top |
|
|
skkp2006
New User
Joined: 14 Jul 2006 Posts: 93 Location: Chennai,India
|
|
|
|
Arun,
I ran a simple test using the below JCL
Code: |
//IEB EXEC PGM=IEBGENER
//SYSOUT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSUT1 DD DSN=input.file,DISP=SHR
//SYSUT2 DD DSN=output.file,DISP=OLD
//SYSIN DD DUMMY |
what i have in sysprint is
Code: |
DATA SET UTILITY - GENERATE
PROCESSING ENDED AT EOD |
and no indication that syncsort is getting invoked.... :-(
i ran the same job with icegener and got the following in sysout
Code: |
DATA SET UTILITY - GENERATE
SYNCSORT COPY FEATURE CALLED - RC = 0 |
i am rather confused now .... what does this indicate? is BetterGener getting invoked when i use ICEGENER or is this a simple sort copy job? and how do i differenciate a BetterGener job with others ?
Syam |
|
Back to top |
|
|
fazilbe
New User
Joined: 03 Dec 2008 Posts: 25 Location: INDIA
|
|
|
|
Hi,
I used the RECFM=FB,LRECL=796 and coded BUFNO=50 in the input and output files. I could see a improvement of over 90% in the CPU time , But very less improvements in the elapsed time. Sort ran for 3 hours and IEBGENER ran for 3hours and 20 minutes. Also the FASTGENER is being used in our system which makes the IEBGENER copy very fast and performance is equivalent to SORT copy.
But the point is, In the original prod run, same job ran for 10 hours. But while testing it with SORT and IEBGENER job it just ran for 3 hours with all the records properly sorted as that of production machine. Not sure why such a difference even while using IEBGENER. Even if the production machine was busy and had so many running at the same time, I think 7 hours variance as some what odd. Does RECFM=FB,LRECL=796 would have contributed to this variance in test regions. Please advise.
Also please find the code below which i used for testing.
//SORTIN DD DSN=TEMP1,
// DISP=SHR,
// BUFNO=50,
// DCB=(LRECL=796,RECFM=FB)
// DD DSN=TEMP2,
// DISP=SHR,
// BUFNO=50,
// DCB=(LRECL=796,RECFM=FB)
//SORTOUT DD DSN=OUTPUT11,
// DISP=(NEW,CATLG,DELETE),
// VOL=(,,,99),
// BUFNO=50,
// UNIT=CART,
// DCB=(LRECL=796,RECFM=FB)
//SYSIN DD *
SORT FIELDS=COPY
Regards,
Fazil |
|
Back to top |
|
|
Alissa Margulies
SYNCSORT Support
Joined: 25 Jul 2007 Posts: 496 Location: USA
|
|
|
|
skkp2006,
Without knowing which of the 2 methods for installing BetterGener was done at your site, it appears that the LINKLIST library containing the entry IEBGENER comes before the IEBGENER alias/ICEGENER alias/SYNCGENR module. Therefore, when you execute IEBGENER, you are getting true IEBGENER. However, when you execute ICEGENER, you are getting the ICEGENER alias for SYNCGENR, which can use SyncSort's COPY feature. |
|
Back to top |
|
|
Douglas Wilder
Active User
Joined: 28 Nov 2006 Posts: 305 Location: Deerfield IL
|
|
|
|
Did you check how long it took the operator to mount the tapes? |
|
Back to top |
|
|
Mistermind
New User
Joined: 08 Feb 2008 Posts: 46 Location: Dublin
|
|
|
|
What is the blocksize in each situation,
the EXCP count,
the TCB count,
the SRC count on the SDSF joblog?
The MIPs of the test and Production cpu's?
Presumably your input is on disk. Try switch to UNIT=SYSDA instead of =CART on the output and watch the difference in peripheral performance. |
|
Back to top |
|
|
fazilbe
New User
Joined: 03 Dec 2008 Posts: 25 Location: INDIA
|
|
|
|
Hi,
I found that difference in the run time between the test machine and prod machine is bound to happen because during the month end, prod machine seems to be very busy and necessary resource would not be available for the jobs at all the time. I learnt this issue on heavy traffic from one of the SME's.
Also the encouraging thing while using SORT is the reduction in CPU time. I saw 90% reduction in CPU time while using SORT. So i have got the approval to use the SORT copy technique for the job i was analysing.
Also could someone please let me know how to find whether the tapes being mounted quickly or being done with some delay?.
I will let you all know the updates on this issue very soon.
Regards,
Fazil |
|
Back to top |
|
|
fazilbe
New User
Joined: 03 Dec 2008 Posts: 25 Location: INDIA
|
|
|
|
Hi Mistermind,
I am afraid i cannot move the output to DASD from tape because of its huge size. Its holding more than 150GB of data and expected to hold the same every month. So we cannot have this amount of data in DASD in production machine which could results in space shortage. Please provide your valuable suggestion on this. Thank you.
Regards,
Fazil |
|
Back to top |
|
|
Debabrata Pruseth
New User
Joined: 11 Dec 2008 Posts: 59 Location: Pune , India
|
|
|
|
Hi fazilbe
Code: |
Description Utility(Minimum CPU time)
VSAM to VSAM ADRDSSU
VSAM to IAM SORT
VSAM to QSAM SORT
IAM to VSAM SORT
IAM to IAM ADRDSSU
IAM to QSAM SORT
QSAM to VSAM SORT
QSAM to IAM SORT
QSAM to QSAM SORT
VSAM to Tape ADRDSSU
IAM to Tape ADRDSSU
QSAM to Tape SORT |
This was found out in a extensive research on backup utilities in one of the projects in our organization . ADRDSSU is the best utility for tape backup . You can have a similar testing in your environment and prepare a table for reference that your can use for future purposes.
You have to take similar size files and run the various utilities on them . Compare the CPU time and cost to pin point the best utility. I think the table above will hold true universally. |
|
Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
|
|
|
|
Hello,
It is good to keep in mind that cpu used is not usually a main consideration when comparing data copy utilities.
Another consideration is whether the copy is to be done on one or a few huge files of the same organization type or if the copy is for possibly hundreds or thousands of unlike files. |
|
Back to top |
|
|
Debabrata Pruseth
New User
Joined: 11 Dec 2008 Posts: 59 Location: Pune , India
|
|
|
|
dick scherrer wrote: |
It is good to keep in mind that cpu used is not usually a main consideration when comparing data copy utilities.
|
The main reason i referred CPU time is
a) Lesser the CPU time lesser the cost associated .
b) If CPU time is less then the batch window decreases . Ideally the critical period of any production batch should be minimum . At least in the system that we use ( Vision Plus for Credit Cards ) we are always advised not to put anything non critical that will increase the critical time in the production batch and have it as low as possible. Small changes such as using ADRDSSU has helped us in decreasing batch window timings in certain project to a great extent.
Can you give me from your experience what are the other main consideration when comparing data copy utilities ( apart from the size and type of data ). This will help us a lot in deciding the right compare utility for the right job . |
|
Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
|
|
|
|
Hello,
Quote: |
Lesser the CPU time lesser the cost associated . |
True in the purest sense, but quite misleading. The vast majority of the cost of high volume copies is the data transfer which reauires very little cpu time.
Yes, often there is a batch window that needs to be protected.
Time permitting, i'll try to add a few more thoughts to this over the weekend. Please post a reply/reminder to this topic Friday night or Sat so i'll get an alert e-mail. If you have the info you want, please post that as well. |
|
Back to top |
|
|
|