IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

BMC unload execution time reduction


IBM Mainframe Forums -> IMS DB/DC
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
ManSab

New User


Joined: 23 Sep 2006
Posts: 13
Location: Pune

PostPosted: Fri Nov 22, 2013 4:34 am
Reply with quote

Need some expert advice on BMC unload execution time reduction:
Have a scenario where I need to change IMS key fields(data not definition). I am using BMC unload utility with an UserExit; where key data is changed. After successful unload with correction, I reload data back. I am able to accomplish on smaller databases.
However for larger databases (approx. 10 million record size), while unloading (with UserExit) the job ran more than 16 hours thrice before I cancelled it. DBA and I tried various options to reduce the execution time but no luck. Options we tried:
- Changed DFSVSAMP parameters (increased IOBF )
- Job class and Region parameter.
- Changed UserExit default BUFSPACE(20) to BUFSPACE(400)
- Few more setting based on BMC recommendation - DFSURGU1(SHORT), EXPAND(NO) ,LPCK(NO)

Unload JCL:
Code:
//UNLOAD    EXEC PGM=DFSRRC00,PARM='ULU,DFSURGU0,XRXDB',
//         REGION=3072K
//IMS      DD  DSN=XXXXXX.DBDLIB,DISP=SHR
//SYSPRINT DD  SYSOUT=*
//DFSVSAMP DD  DSN=XXXXXX.XXXXX.MACLIB(XRVSAMP),DISP=SHR
//DFSURGU1 DD  DSN= XXXXXX.XXXXX.XRXRIDUN.SS301,
//             UNIT=(TAPE,,DEFER),DCB=BUFNO=5,
//             LABEL=(4,,,RETPD=0060),
//             DISP=(NEW,CATLG,KEEP),
//             VOLUME=(,RETAIN)
//DFSURGU2 DD  DSN= XXXXXX.XXXXX.XRXRIDUN.SS302,
//             UNIT=(TAPE,,DEFER),DCB=BUFNO=5,
//             LABEL=4,
//             DISP=(NEW,CATLG,KEEP),
//             VOLUME=(,RETAIN)
 //XRXDB    DD  DSN= XXXXXX.XXXXX.SS300.B4,
//             DISP=(OLD,KEEP,KEEP)
//XRXDB   DD  DSN= XXXXXX.XXXXX.SS00I.B4,
//             DISP=(OLD,KEEP,KEEP)
//PLUSIN DD *
UNLOAD USEREXIT(MDCTAK1) BUFSPACE(400) DFSURGU1(SHORT) EXPAND(NO) -
       LPCK(NO)
//*

Appreciate your advice ? - Thanks
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Fri Nov 22, 2013 8:10 pm
Reply with quote

Hello,

How many output tape volumes were created?

What was the wait time between mounts?

No unload of only 10mil records should take anywhere near that long.
Back to top
View user's profile Send private message
Ed Goodman

Active Member


Joined: 08 Jun 2011
Posts: 556
Location: USA

PostPosted: Fri Nov 22, 2013 8:40 pm
Reply with quote

How long does it take without the exit routine?

I'm asking because of a problem I had MAAAANNNNYYYY years ago doing something similar. (right after 9/11)



In my case, we neglected to make the exit routine program resident. So each segment passed to it would re-load the damn thing. It was taking forever too.

Here are the highlights from the compile JCL I made to make this work. It was COBOL II, and I'm certain there is a better way.

I had to rebuild the IGZEOPT module with reentrant/reuse (or something like that)


Code:

//CMP     EXEC PGM=IGYCRCTL, REGION=1024K               
//   PARM=('BUF(32760),ADV,RENT,DATA(31),FDUMP,F(I)',   
//   ',NONUM,OFF',                                     
//   ',NOCMPR2,SEQ,NUMPROC(PFD),NOSSR,RES,NODYNAM')     

//MYLIB     DD DSN=WLF6.WNEG.LOAD,DISP=SHR       

//SYSLIN    DD DSN=&&LOADSET,DISP=(OLD,DELETE)     
//          DD DSN=WLF6.WNEG.JCL(ASMINCL),DISP=SHR 

 INCLUDE  MYLIB(IGZEOPT) 



I know that we ended up creating a new SCLM type for these exits, and that's what I always use now.

I'm hoping that someone here can tell you a better way to keep a program loaded.
Back to top
View user's profile Send private message
Ed Goodman

Active Member


Joined: 08 Jun 2011
Posts: 556
Location: USA

PostPosted: Fri Nov 22, 2013 9:14 pm
Reply with quote

This angered me that I couldn't remember...so I went hunting!

I found the SCLM hooks that are in use with Enterprise COBOL

Code:

Member WTSO.XXXX.SOURCE(CEEUOPT) has:
CEEUOPT  CSECT                             
CEEUOPT  AMODE ANY                         
CEEUOPT  RMODE ANY                         
         CEEXOPT RTEREUS=(ON),             
               TRAP=(OFF)                 
         END                               

Assembled into
WTSO.XXXX.LOAD(CEEUOPT)

The SCLM language macro for the Link Edit has:
BMCLE370 FLMSYSLB   WTSO.XXXX.LOAD                             
*                                                               
         FLMLANGL    LANG=BMCLE370,CANEDIT=N,VERSION=L370V1.0   
          FLMTRNSL   CALLNAM='BMCLKED/370',                     
               FUNCTN=BUILD,                                     
               COMPILE=HEWL,                                     
               VERSION=F64,                                     
               GOODRC=4,                                         
               PORDER=3,                                         
               OPTIONS=(MAP,XREF,LIST,SIZE=(256K,48K),LET,REUS) 

*                                                               
*  1       (* SYSLIN *)                                         
           FLMALLOC  IOTYPE=S,KEYREF=SINC,RECFM=FB,LRECL=80,   
               RECNUM=190000,DDNAME=SYSLIN                     
            FLMCPYLB WTSO.XXXX.LOAD                           
*                                                               
*  2       (* LOAD MODULE NAME *)                               
           FLMALLOC  IOTYPE=L,KEYREF=LOAD                       
            FLMCPYLB WTSO.XXXX.LOAD                           
*                                                               
*  3       (* SYSLMOD *)                                       
           FLMALLOC  IOTYPE=P,KEYREF=LOAD,RECFM=U,LRECL=0,     
               RECNUM=190000,DIRBLKS=40,BLKSIZE=6144           
            FLMCPYLB WTSO.XXXX.LOAD                           
*                                                               
*  4       (* SYSLIB  *)                                       
           FLMALLOC  IOTYPE=A,DDNAME=SYSLIB                     
            FLMCPYLB WTSO.XXXX.LOAD                           
            FLMCPYLB SYS1.SCEELKED                             
            FLMCPYLB SYS1.SIGYCOMP             
            FLMCPYLB SYS1.LINKLIB             

The SCLM language parser (Compile) has:
          FLMTRNSL   CALLNAM='BMCEXIT',                         
               FUNCTN=BUILD,                                     
               DSNAME=SYS1.SIGYCOMP,                             
               COMPILE=IGYCRCTL,                                 
               VERSION=1.0,                                     
               GOODRC=4,                                         
               PORDER=3,                                         
               OPTIONS=(APOST,C,DATA(31),NODYN,LIB,NORENT,XREF, 
               OFF,MAP,NUMPROC(PFD),NOCMPR2,FLAG(I,W),           
               NOADV,TEST(NONE,SYM),LANGUAGE(EN),SEQ,BUF(32760))



I also realized that we have a vestigial line of code in all of our exits. At the start of the Procedure Division:
IF (FIRST-TIME-SWITCH EQUAL SPACES)
MOVE HIGH-VALUES TO FIRST-TIME-SWITCH
DISPLAY 'START DB LOAD'

This was added by me to show that the program only loaded once. You could add something similar to prove you have it working.

If it's not, you see that display statement for every input segment.
Back to top
View user's profile Send private message
ManSab

New User


Joined: 23 Sep 2006
Posts: 13
Location: Pune

PostPosted: Fri Nov 22, 2013 10:09 pm
Reply with quote

Ed Goodman :
- UserExit gets called, I did check with display statement including DISPLAY 'START DB LOAD' . Had to remove displays because of additional overhead and to avoid operations team questions.
- Unload without userexit completes in 60 sec in prod and in 80 seconds in test (approx).
- You may be spot on with program not being resident. I will check on this.
dick scherrer:
I don't see multiple volumes but one in job log. I will check with storage support to confirm.


Will keep you posted.... - Thanks
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Fri Nov 22, 2013 10:23 pm
Reply with quote

Hello,

I believe you've shown that the userexit is the culprit.

Work with someone on your system who provides performance measurement support and see how much of the time is in the exit code (cpu or i/o or both).

As Ed mentioned, constantly loading a module over and over will cause a Lot of unneeded overhead.

Originally, i thought the time it took to mount tapes might be a problem. We can forget that as only one volume is needed icon_cool.gif
Back to top
View user's profile Send private message
ManSab

New User


Joined: 23 Sep 2006
Posts: 13
Location: Pune

PostPosted: Fri Nov 22, 2013 10:38 pm
Reply with quote

Ed Goodman and dick scherrer:
I debugged the unload step (userexit), and it does get loaded every time segment is passed on to it. Will compile with these parameters now:
Code:
//COMPX   EXEC PGM=IGYCRCTL,REGION=7M,
//        PARM=(APOST,DYNAM,FLAG(W,E),LIB,MAP,NOSEQ,OFFSET,
//        RENT,SSRANGE,SOURCE,XREF,VBREF,DATA(31),NOSSR,RES)


Thanks
Back to top
View user's profile Send private message
ManSab

New User


Joined: 23 Sep 2006
Posts: 13
Location: Pune

PostPosted: Sat Nov 23, 2013 12:50 am
Reply with quote

with compile parameter changes, job still running from 2 hours..
Ed Goodman :
Can you please elaborate this :
Quote:

I had to rebuild the IGZEOPT module with reentrant/reuse (or something like that)
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Sat Nov 23, 2013 1:01 am
Reply with quote

Hello,

Might there be a parameter specific to BMC to tell the run Not to reload every time?

BMC support should be able to tell you how to proceed to get around all of these loads.
Back to top
View user's profile Send private message
Ed Goodman

Active Member


Joined: 08 Jun 2011
Posts: 556
Location: USA

PostPosted: Sat Nov 23, 2013 1:51 am
Reply with quote

Like I said, this was 11 years ago. BMC may have changed something by now. I just remember that they couldn't help us then, and that the instructions/hints to change the compile were in the BMC manual.

Mansab, just changing the compile will NOT work. You also have to work with the link editor. The item you asked me to elaborate on is about changing the options that get used during link-edit.

If you are on Enterprise COBOL, you'll want to use what's in my second post dealing with the CEE version.
Back to top
View user's profile Send private message
Ed Goodman

Active Member


Joined: 08 Jun 2011
Posts: 556
Location: USA

PostPosted: Sat Nov 23, 2013 1:53 am
Reply with quote

From my current BMC manual:
You may receive positive results by using NORENT and NODYNAM as compile options.
When using a COBOL/370 exit with the Unload function or the Reload function, you
may need to assemble an application-specific copy of module CEEUOPT with
RTEREUS(ON) and TRAP=OFF. If your current LE/370 run-time keywords module
specifies RTEREUS(OFF) and TRAP=ON, you can assemble a CEEUOPT module with
RTEREUS(ON) and TRAP=OFF to another library and INCLUDE it in the link-edit of your
user exit. The CEEUOPT module must be assembled with AMODE(31). For details
about the LE environment, see the IBM Language Environment manuals.
Back to top
View user's profile Send private message
ManSab

New User


Joined: 23 Sep 2006
Posts: 13
Location: Pune

PostPosted: Mon Nov 25, 2013 10:39 am
Reply with quote

So this is how it crumbled down...........

per Ed advice and BMC Manual

1) Assembled and link/edit CEEUOPT parameter changes into a different LOADLIB.
Code:

//SYSIN  DD *
  CEEUOPT  CSECT
  CEEUOPT  AMODE 31
  CEEUOPT  RMODE ANY
  CEEXOPT  RTEREUS=(ON),
                 TRAP=(OFF,NOSPIE)
  END
/*

2) Included above CEEUOPT load during link-edit of the UserExit.
Code:

//LINK      EXEC PGM=IEWL,
//          PARM=(LIST,XREF,REUS,NORENT,AMODE(31),RMODE(ANY))
//SYSPRINT  DD  SYSOUT=*
//SYSTERM   DD  SYSOUT=*
//SYSLMOD   DD  DSN=XXXX.XXXX.LINKLIB,DISP=SHR
//SYSLIN    DD  DSN=&&COMP1,DISP=(OLD,PASS)
//          DD  *
    INCLUDE SYSLIB(CEEUOPT) 
    NAME USEXCV80(R)
//SYSLIB    DD  DSN= XXXX. CEEUOPT.LOADLIB,DISP=SHR

3) Submitted unload job for 10 million records, and guess what it completed in 20 SECONDS.


Thanks much Ed and Dick, you guys are awesome..!!!
Back to top
View user's profile Send private message
Ed Goodman

Active Member


Joined: 08 Jun 2011
Posts: 556
Location: USA

PostPosted: Mon Nov 25, 2013 9:10 pm
Reply with quote

Can I get WOOT WOOT!?!?!

Seriously, how good do you feel right now? It's awesome when it works!
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Mon Nov 25, 2013 10:20 pm
Reply with quote

What was the Hannibal Smith saying:
I love it when a plan comes together"

Good to hear it is working - thank you for letting us know icon_smile.gif

Oh, yeah, aaand:

WOOT WOOT!?!?!

d
Back to top
View user's profile Send private message
Ed Goodman

Active Member


Joined: 08 Jun 2011
Posts: 556
Location: USA

PostPosted: Tue Nov 26, 2013 2:45 am
Reply with quote

Now, the question is, Is there a way to use 'RTEREUS=(ON)' for a single link-edit? Or does it have to be part of the CEEUOPT member?
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> IMS DB/DC

 


Similar Topics
Topic Forum Replies
No new posts how to eliminate null indicator value... DB2 7
No new posts Unload and Load ISPF Table TSO/ISPF 4
No new posts Load new table with Old unload - DB2 DB2 6
No new posts To get the the current time DFSORT/ICETOOL 13
No new posts RC query -Time column CA Products 3
Search our Forums:

Back to Top