IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

IEBGENER vs SORT


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
fazilbe

New User


Joined: 03 Dec 2008
Posts: 25
Location: INDIA

PostPosted: Tue Jan 20, 2009 6:31 pm
Reply with quote

Hi,

I have a job which runs for close to 10 hours each month. When analyzed found that one particular PROC step consuming all the run time. The step just does a copy of a huge file to a GDG using IEBGENER. Input file has a more than 100gb of record. Can we make this step more faster using any other technique. Will SORT be faster compared to IEBGENER. Please help me solve this issue. Please find the JCL below.
Code:

//JS010    EXEC AIP04000,             
//             RETPD='720',         
//             UNIT=CART,           
//             UNIT1='SYSDA',       
//             SPACEMY='100,50',           
//PS040    EXEC PGM=IEBGENER,                           
//             COND=(0,LT)                               
//SYSUT1   DD  DSN=temp1,       
//             DISP=SHR                                                     
//SYSUT2   DD  DSN=temp2(+1),
//             VOL=(,,,99),                                 
//             FREE=CLOSE,                                 
//             DISP=(NEW,CATLG,DELETE),                     
//             UNIT=&UNIT,                                 
//             LABEL=RETPD=&RETPD,                         
//             DCB=(&DSCB)                                 
//SYSIN    DD  DUMMY                                       
//SYSPRINT DD  SYSOUT=*                                 
//SYSUDUMP DD  SYSOUT=Y

Regards,
Fazil
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA

PostPosted: Tue Jan 20, 2009 6:38 pm
Reply with quote

Add buffers to input and output file; with large files you should never use the z/OS default.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales

PostPosted: Tue Jan 20, 2009 6:50 pm
Reply with quote

Quote:
Will SORT be faster compared to IEBGENER. Please help me solve this issue. Please find the JCL below.

May I suggest that you try it for yourself. The workload of your machine can not be replicated on ours, and this is one of the major factors involved.

Code:
//             DCB=(&DSCB)

What are the specified paramters for &DSCB and to they allow for SDB to be used.
Back to top
View user's profile Send private message
fazilbe

New User


Joined: 03 Dec 2008
Posts: 25
Location: INDIA

PostPosted: Tue Jan 20, 2009 7:04 pm
Reply with quote

Hi,

DCB=(796 27,860 0 FB) is what i could find in the spool log. Does this help?. Please let me know if more information is needed.

Regards,
Fazil
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales

PostPosted: Tue Jan 20, 2009 7:29 pm
Reply with quote

I would replace that line with
Code:
//      RECFM=FB,LRECL=796


I assume that the output is tape and the code above will allow the system to determine the best output blocksize. This would allow more records per block to be written so less writes are required.

Is the input file tape also ? If so, is the blocksize optimum for tape or also defined for DASD ?

Also look at the suggestion by Robert to increase the I/O buffering, say 20 buffers per DD may significantly improve performance.
Back to top
View user's profile Send private message
Bill Dennis

Active Member


Joined: 17 Aug 2007
Posts: 562
Location: Iowa, USA

PostPosted: Tue Jan 20, 2009 8:02 pm
Reply with quote

Another option would be to allocate multiple tape drives on input and output as
Code:
//  UNIT=(&UNIT,2)


This is less effective on newer tape technology where the "end" is back at the "start" but if your tapes are virtual and are being recalled it might help.
Back to top
View user's profile Send private message
Douglas Wilder

Active User


Joined: 28 Nov 2006
Posts: 305
Location: Deerfield IL

PostPosted: Tue Jan 20, 2009 9:01 pm
Reply with quote

Check how long it was between the time the system requested the tape to be mounted and the time the operator informed the system that the tape was mounted. If the operators do not mount the tapes in a timely manor no system changes can improve the time that much.
Do allow the system to choose the best block size and increate the buffers as stated by others, and change IEBGENER to ICEGENER (If it has been installed this way on your system). This will use sort to copy the file and my experience is that this will help quite a bit.
Back to top
View user's profile Send private message
Alissa Margulies

SYNCSORT Support


Joined: 25 Jul 2007
Posts: 496
Location: USA

PostPosted: Tue Jan 20, 2009 9:44 pm
Reply with quote

If you are running SyncSort for z/OS, verify that the BetterGener function has been installed. The BetterGener utility program offers improved performance over IEBGENER in straight copy tasks and is transparent to users. Implementing BetterGener will provide significant savings in system resources such as elapsed time, CPU time, and channel utilization through improved I/O handling. Once BetterGener is activated, eligible IEBGENER job streams are automatically processed by SyncSort. There is no need to replace working IEBGENER job streams with explicit requests for SyncSort's more efficient copying.
Back to top
View user's profile Send private message
Frank Yaeger

DFSORT Developer


Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA

PostPosted: Tue Jan 20, 2009 10:11 pm
Reply with quote

Quote:
Do allow the system to choose the best block size and increate the buffers as stated by others, and change IEBGENER to ICEGENER (If it has been installed this way on your system). This will use sort to copy the file and my experience is that this will help quite a bit.


If DFSORT's ICEGENER has been installed as a replacement for IEBGENER, then you don't need to change PGM=IEBGENER to PGM=ICEGENER. ICEGENER will be used automatically. If DFSORT's ICEGENER has not been installed as a replacement for IEBGENER, then using PGM=ICEGENER would be necessary.

You don't need to increase the buffers for ICEGENER - it does its own optimal buffering.

For more information on DFSORT's ICEGENER, see:

publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/ICE1CA30/9.2.16?DT=20080528171007
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia

PostPosted: Wed Jan 21, 2009 5:21 am
Reply with quote

Hi,

I would also check the BLOCKSIZE of the input file.


Gerry
Back to top
View user's profile Send private message
fazilbe

New User


Joined: 03 Dec 2008
Posts: 25
Location: INDIA

PostPosted: Wed Jan 21, 2009 1:06 pm
Reply with quote

Hi,

I will try the valuable suggestions given by everybody and let you all know the outcome of it. Thanks a lot.

Regards,
Fazil
Back to top
View user's profile Send private message
fazilbe

New User


Joined: 03 Dec 2008
Posts: 25
Location: INDIA

PostPosted: Wed Jan 21, 2009 3:14 pm
Reply with quote

Hi,

Actually input is in DASD and the output GDG has been written to TAPE.
Back to top
View user's profile Send private message
skkp2006

New User


Joined: 14 Jul 2006
Posts: 93
Location: Chennai,India

PostPosted: Fri Jan 23, 2009 8:17 am
Reply with quote

Alissa,

Quote:
"If you are running SyncSort for z/OS, verify that the BetterGener function has been installed. "


How can we possibly check whether BetterGener has been installed in our shop? Does it come along with SyncSort or is it a pay and use product?


Syam
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2481
Location: @my desk

PostPosted: Fri Jan 23, 2009 9:59 am
Reply with quote

Quote:
Once BetterGener is activated, eligible IEBGENER job streams are automatically processed by SyncSort
skkp2006,

As pointed out by Alissa, you can run a small IEBGENER copy and check if SyncSort gets invoked.
Back to top
View user's profile Send private message
skkp2006

New User


Joined: 14 Jul 2006
Posts: 93
Location: Chennai,India

PostPosted: Fri Jan 23, 2009 10:22 am
Reply with quote

Arun,


I ran a simple test using the below JCL

Code:
//IEB      EXEC PGM=IEBGENER
//SYSOUT   DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSUT1   DD DSN=input.file,DISP=SHR
//SYSUT2   DD DSN=output.file,DISP=OLD
//SYSIN    DD DUMMY




what i have in sysprint is

Code:
DATA SET UTILITY - GENERATE
PROCESSING ENDED AT EOD



and no indication that syncsort is getting invoked.... :-(



i ran the same job with icegener and got the following in sysout


Code:
DATA SET UTILITY - GENERATE
SYNCSORT COPY FEATURE CALLED - RC = 0




i am rather confused now icon_rolleyes.gif .... what does this indicate? is BetterGener getting invoked when i use ICEGENER or is this a simple sort copy job? and how do i differenciate a BetterGener job with others ?



Syam
Back to top
View user's profile Send private message
fazilbe

New User


Joined: 03 Dec 2008
Posts: 25
Location: INDIA

PostPosted: Fri Jan 23, 2009 7:06 pm
Reply with quote

Hi,

I used the RECFM=FB,LRECL=796 and coded BUFNO=50 in the input and output files. I could see a improvement of over 90% in the CPU time , But very less improvements in the elapsed time. Sort ran for 3 hours and IEBGENER ran for 3hours and 20 minutes. Also the FASTGENER is being used in our system which makes the IEBGENER copy very fast and performance is equivalent to SORT copy.

But the point is, In the original prod run, same job ran for 10 hours. But while testing it with SORT and IEBGENER job it just ran for 3 hours with all the records properly sorted as that of production machine. Not sure why such a difference even while using IEBGENER. Even if the production machine was busy and had so many running at the same time, I think 7 hours variance as some what odd. Does RECFM=FB,LRECL=796 would have contributed to this variance in test regions. Please advise.

Also please find the code below which i used for testing.

//SORTIN DD DSN=TEMP1,
// DISP=SHR,
// BUFNO=50,
// DCB=(LRECL=796,RECFM=FB)
// DD DSN=TEMP2,
// DISP=SHR,
// BUFNO=50,
// DCB=(LRECL=796,RECFM=FB)
//SORTOUT DD DSN=OUTPUT11,
// DISP=(NEW,CATLG,DELETE),
// VOL=(,,,99),
// BUFNO=50,
// UNIT=CART,
// DCB=(LRECL=796,RECFM=FB)
//SYSIN DD *
SORT FIELDS=COPY

Regards,
Fazil
Back to top
View user's profile Send private message
Alissa Margulies

SYNCSORT Support


Joined: 25 Jul 2007
Posts: 496
Location: USA

PostPosted: Fri Jan 23, 2009 9:41 pm
Reply with quote

skkp2006,

Without knowing which of the 2 methods for installing BetterGener was done at your site, it appears that the LINKLIST library containing the entry IEBGENER comes before the IEBGENER alias/ICEGENER alias/SYNCGENR module. Therefore, when you execute IEBGENER, you are getting true IEBGENER. However, when you execute ICEGENER, you are getting the ICEGENER alias for SYNCGENR, which can use SyncSort's COPY feature.
Back to top
View user's profile Send private message
Douglas Wilder

Active User


Joined: 28 Nov 2006
Posts: 305
Location: Deerfield IL

PostPosted: Fri Jan 23, 2009 9:47 pm
Reply with quote

Did you check how long it took the operator to mount the tapes?
Back to top
View user's profile Send private message
Mistermind

New User


Joined: 08 Feb 2008
Posts: 46
Location: Dublin

PostPosted: Fri Jan 30, 2009 7:56 pm
Reply with quote

What is the blocksize in each situation,
the EXCP count,
the TCB count,
the SRC count on the SDSF joblog?
The MIPs of the test and Production cpu's?

Presumably your input is on disk. Try switch to UNIT=SYSDA instead of =CART on the output and watch the difference in peripheral performance.
Back to top
View user's profile Send private message
fazilbe

New User


Joined: 03 Dec 2008
Posts: 25
Location: INDIA

PostPosted: Wed Feb 04, 2009 12:24 pm
Reply with quote

Hi,

I found that difference in the run time between the test machine and prod machine is bound to happen because during the month end, prod machine seems to be very busy and necessary resource would not be available for the jobs at all the time. I learnt this issue on heavy traffic from one of the SME's.

Also the encouraging thing while using SORT is the reduction in CPU time. I saw 90% reduction in CPU time while using SORT. So i have got the approval to use the SORT copy technique for the job i was analysing.

Also could someone please let me know how to find whether the tapes being mounted quickly or being done with some delay?.

I will let you all know the updates on this issue very soon.

Regards,
Fazil
Back to top
View user's profile Send private message
fazilbe

New User


Joined: 03 Dec 2008
Posts: 25
Location: INDIA

PostPosted: Wed Feb 04, 2009 12:28 pm
Reply with quote

Hi Mistermind,

I am afraid i cannot move the output to DASD from tape because of its huge size. Its holding more than 150GB of data and expected to hold the same every month. So we cannot have this amount of data in DASD in production machine which could results in space shortage. Please provide your valuable suggestion on this. Thank you.

Regards,
Fazil
Back to top
View user's profile Send private message
Debabrata Pruseth

New User


Joined: 11 Dec 2008
Posts: 59
Location: Pune , India

PostPosted: Wed Feb 04, 2009 5:46 pm
Reply with quote

Hi fazilbe

Code:
Description              Utility(Minimum CPU time)

VSAM to VSAM                  ADRDSSU
VSAM to IAM                   SORT
VSAM to QSAM                  SORT
IAM  to VSAM                  SORT
IAM  to IAM                   ADRDSSU
IAM  to QSAM                  SORT
QSAM to VSAM                  SORT
QSAM to IAM                   SORT
QSAM to QSAM                  SORT
VSAM to Tape                  ADRDSSU
IAM  to Tape                  ADRDSSU
QSAM to Tape                  SORT


This was found out in a extensive research on backup utilities in one of the projects in our organization . ADRDSSU is the best utility for tape backup . You can have a similar testing in your environment and prepare a table for reference that your can use for future purposes.

You have to take similar size files and run the various utilities on them . Compare the CPU time and cost to pin point the best utility. I think the table above will hold true universally.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Thu Feb 05, 2009 3:03 am
Reply with quote

Hello,

It is good to keep in mind that cpu used is not usually a main consideration when comparing data copy utilities.

Another consideration is whether the copy is to be done on one or a few huge files of the same organization type or if the copy is for possibly hundreds or thousands of unlike files.
Back to top
View user's profile Send private message
Debabrata Pruseth

New User


Joined: 11 Dec 2008
Posts: 59
Location: Pune , India

PostPosted: Thu Feb 05, 2009 11:19 am
Reply with quote

dick scherrer wrote:


It is good to keep in mind that cpu used is not usually a main consideration when comparing data copy utilities.



The main reason i referred CPU time is

a) Lesser the CPU time lesser the cost associated .
b) If CPU time is less then the batch window decreases . Ideally the critical period of any production batch should be minimum . At least in the system that we use ( Vision Plus for Credit Cards ) we are always advised not to put anything non critical that will increase the critical time in the production batch and have it as low as possible. Small changes such as using ADRDSSU has helped us in decreasing batch window timings in certain project to a great extent.

Can you give me from your experience what are the other main consideration when comparing data copy utilities ( apart from the size and type of data ). This will help us a lot in deciding the right compare utility for the right job .
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Thu Feb 05, 2009 12:39 pm
Reply with quote

Hello,

Quote:
Lesser the CPU time lesser the cost associated .
True in the purest sense, but quite misleading. The vast majority of the cost of high volume copies is the data transfer which reauires very little cpu time.

Yes, often there is a batch window that needs to be protected.

Time permitting, i'll try to add a few more thoughts to this over the weekend. Please post a reply/reminder to this topic Friday night or Sat so i'll get an alert e-mail. If you have the info you want, please post that as well.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM Goto page 1, 2  Next

 


Similar Topics
Topic Forum Replies
No new posts How to split large record length file... DFSORT/ICETOOL 8
No new posts JCL sort card - get first day and las... JCL & VSAM 9
No new posts Sort First/last record of a subset th... DFSORT/ICETOOL 7
No new posts how to calculate SUM value for VB fil... DFSORT/ICETOOL 1
No new posts how to calculate SUM for VB file usin... JCL & VSAM 1
Search our Forums:

Back to Top