IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Performance Batch Process


IBM Mainframe Forums -> COBOL Programming
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
insomniaa

New User


Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes

PostPosted: Tue Aug 14, 2007 8:30 pm
Reply with quote

Good afternoon all!!!

We are running a process that is having serious issues with performance and volume of information generated and wanted some feedback from you experts in batch processes:

1. How can repetitive call affect the performance of a batch program. Our process will at least make 2 million calls to the same cobol routine.

2. Our process currently generates a file with more than 130 million records that a filtered in the next step of the process (confrontation of two files). Would it be better to do everything in the same program bering in mind that the filter file has no more than 200 records. I mean, is it better a search all than a file confrontation?

Thanks!!!!
Leo
Back to top
View user's profile Send private message
William Thompson

Global Moderator


Joined: 18 Nov 2006
Posts: 3156
Location: Tucson AZ

PostPosted: Tue Aug 14, 2007 8:47 pm
Reply with quote

Not quite sure of what you are asking, but...
Best would have the big file and the little file in the same matching key sequence - time spent would be almost all just reading and processing the selected data and very little matching.
Second best (if the big file can't be sorted) would be having the little file as binary searchable in-core table.

At 130 million repititions, even the adding of an increment to a ZD counter changed to a PD counter would help. And I'd sure look for others such as that.....Unsigned variables, even digit PD, extra subscript/index calculations......
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Tue Aug 14, 2007 8:48 pm
Reply with quote

Hello,

Please define "serious performance issues".

Searching in the first program while passing the 130mil records should be much faster than making another pass of the 130mil records.

If the 200 values were sorted before being loaded into an array, you could use a "SEARCH ALL" (binary search) and run even more efficiently. If you build a 200 entry array regardless of sequential or binary search, it is good practice to look at the "current" value and compare it to the "previous" value and if they are equal, do not search the array again as the entry in the array is already located.

Make sure the called module is reusable and not reloaded 2mil times.
Back to top
View user's profile Send private message
insomniaa

New User


Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes

PostPosted: Tue Aug 14, 2007 9:07 pm
Reply with quote

Thanks for the answer but not quite what I was looking for.

Our serious issues means that we have 10 JCL's doing the same logic (sweep process of customer database against a blacklist):

1. Each of the JCL's are or will be creating a file with at least 130 million records.
2.The process seems that it will be running for more than 5 hours.... never tested in our Preproduction env. but in Development it had ran for 17 hours and never finished because I cancelled it before.

The batch program performs a call to a cobol routine per read customer. If we have 2 million customers in the input file it will perform at least 2 million calls. Is this bad for the program performance?

In order to reduce the volume we thought about moving the filtering process into the same batch program, loading the filtering file into the working storage and perform a search all everytime is needed. Is this better, from a performance point of view to perform the filtering process in a different batch program?

I hope that this time is more understandable my enquiry... Cheer!!!
Back to top
View user's profile Send private message
insomniaa

New User


Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes

PostPosted: Tue Aug 14, 2007 9:20 pm
Reply with quote

Ummm.... so if I move the filter process to the same batch program I will reduce time and volume but I still need to reduce even more the execution time of the first batch program (the one that performs the sweep and call 2 million times to the same routine).

I am not sure if the routine is reusable, can i check this anywhere? our system is very limited and we don't have much information, my guess is that it is reusable but it is only a guess.

Once again, thank you for your help and time!!!!
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Tue Aug 14, 2007 9:40 pm
Reply with quote

You're welcome icon_smile.gif

Quote:
Is this better, from a performance point of view to perform the filtering process in a different batch program?


I believe that if you load the 200 entry array and search it in the first set of code, you will perform better than if the process requires another step to pass all of the records again.

Is the module invoked via a static call or a dynamic call? If you use a static call and specify NODYNAM, your process should re-invoke the module without any interaction with the load ibrary. You need to ensure that the code initializes any working variables as they will not be reset on calls 2 thru n.
Back to top
View user's profile Send private message
insomniaa

New User


Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes

PostPosted: Tue Aug 14, 2007 9:56 pm
Reply with quote

Hi, thanks icon_biggrin.gif

The module is invoked via a REXX function and it seems to be a dynamic call. In other words, the batch program invokes a rexx function to call the module.

Will the performance be better if we the code within the module is embedded directly into the batch program? In other words, we get rid of the module.

Leo

icon_lol.gif Muchas gracias!!!
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Tue Aug 14, 2007 10:00 pm
Reply with quote

De Nada,

What service does the rexx "in the middle" provide?

If i could, i would have the "first" program call the sub-program directly and leave out the rexx in the middle.
Back to top
View user's profile Send private message
insomniaa

New User


Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes

PostPosted: Tue Aug 14, 2007 10:09 pm
Reply with quote

Dont know what the REXX is doing, i have tried to understand it but since I dont have experiece with REXX programming... but it seems that is the REXX the one invoking the module....

It is possible to make the first program to call the sub-program directly. Would this improve the execution time?

I can copy you the REXX function if you want....

Thanks!!!
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Tue Aug 14, 2007 10:36 pm
Reply with quote

Hello,

Quote:
It is possible to make the first program to call the sub-program directly. Would this improve the execution time?

I can copy you the REXX function if you want....


I suspect it would improve the execution time if the main code directly called the sub-program - much depends on the "value added" by invoking the rexx code. How does the COBOL code invoke this rexx? Posting the COBOL "setup" for the call and the actual call may help us understand.

Sure, if you post the rexx code (please use the Code tag at the top of the replay panel for both the COBOL and rexx code), we can take a look at it.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Tue Aug 14, 2007 11:12 pm
Reply with quote

If a dynamically called module does not contain the INITIAL statement, and it is not CANCELed, subsequent calls will find working-storage in a last used state.
Back to top
View user's profile Send private message
TG Murphy

Active User


Joined: 23 Mar 2007
Posts: 148
Location: Ottawa Canada

PostPosted: Tue Aug 14, 2007 11:38 pm
Reply with quote

My bet is that Rexx is your problem. As Dick suggested, show us the Rexx code so we can see whether the Rexx program is performing a crucial function - and if it is - whether this function can be carried out from within your COBOL program.

I don't think your COBOL sub-program by itself is your problem - provided you have made it serially reusable using the old fashioned way. The old fashioned way is to only use VALUE statements for defining constants - all variables should be initialized using MOVE, SET, etc..

As DBZ mentioned, you can make your program serially reusable using INITIAL or CANCEL but that's not the correct way to do it - you will suffer a performance hit using INITIAL and CANCEL.
Back to top
View user's profile Send private message
insomniaa

New User


Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes

PostPosted: Wed Aug 15, 2007 1:48 pm
Reply with quote

Hi all... many thanks for your comments. Our process has been running for 19 hours and going.... and we are talking of only one job when we actually have 10 jobs doing the same but with differents customers partitions....

This is the REXX function.
Code:

/* REXX */                                                                 
ARG FX1VEZ FXNAME FXNUM LINE                                               
IF LINE = '@MACRO@'  THEN DO                                               
  QUEUE " EXEC-FUN "FXNAME" W-RUTINA USING W-PARAM1                    "   
  QUEUE "                                  W-PARAM2                    "   
  QUEUE "                                  ........                    "   
  QUEUE "                                  W-PARAMN                    "   
  QUEUE "           END-FUN                                            "   
  QUEUE "* Permite la llamada a rutinas desde programas Mixtos,        "   
  QUEUE "* eliminado areas DFHEIBLK y DFHCOMMAREA para el proceso BATCH"   
  QUEUE "* y dejando los parametros tal y como los recibe para el      "   
  QUEUE "* proceso CICS.                                               "   
  QUEUE "*                                                             "   
  QUEUE "* W-RUTINA    : Rutina a llamar.                              "   
  QUEUE "* W-PARAM1..N : Parametros de la rutina.                      "   
  RETURN 0                                                                 
  END                                                                     
ELSE DO                                                                   
  PULL PARM                                                               
  PULL PARM                                               
  RUTINA=WORD(LINE,2)                                     
  VUSING=''                                               
  IF SUBSTR(WORD(LINE,3),1,5)¬='USING' THEN  DO           
     QUEUE "E*  FALTA USING ";  EXIT 8                   
  END                                                     
                                                         
  QUEUE "P     CALL "RUTINA" USING "                     
  DO I=4 TO WORDS(LINE)-1                                 
     IF POS('CICS',PARM) = 0 THEN DO                     
        IF WORD(LINE,I) = 'DFHEIBLK'    |,               
           WORD(LINE,I) = 'DFHCOMMAREA' THEN ITERATE     
     END                                                 
     QUEUE "P          "WORD(LINE,I)                     
  END                                                     
                                                         
END                                                       
RETURN 0                                                 


And this is how the main program invokes the REXX function:
Code:

INITIALIZE BL25-SALIDA                             
           REPLACING ALPHANUMERIC DATA BY SPACES. 
                                                   
MOVE  ENT1-FIRST-NM       TO BL25-FIRST-1         
MOVE  ENT1-MIDDL-NM       TO BL25-MIDDLE-1         
MOVE  ENT1-LAST-NM        TO BL25-SURNAME-1       
MOVE  ENT1-PAISRESI       TO BL25-PAIS-RESI-1     
MOVE  ENT1-PAISNAC        TO BL25-PAIS-NACI-1     
MOVE  ENT1-CPAIS2NA       TO BL25-PAIS-2NAC-1     
MOVE  ENT1-FECHNACI       TO BL25-FECH-NACI-1     
                                                   
EXEC-FUN                                           
     XX_CALL CA-BHCBL25 USING DFHEIBLK             
                              DFHCOMMAREA         
                              BL25-PARAMETROS     
END-FUN                                           


Hope this helps...

Thanks again!!!

Leo
Back to top
View user's profile Send private message
TG Murphy

Active User


Joined: 23 Mar 2007
Posts: 148
Location: Ottawa Canada

PostPosted: Wed Aug 15, 2007 7:24 pm
Reply with quote

Looks to me like Rexx was used so as to take advantage of its great text parsing abilities. In most cases - a reasonable thing to do.

However, the parsing carried out by the Rexx program could be done using COBOL code - although not nearly as easily. I would suggest this:

Start by modifying your program so that it does NOT call the REXX program. You may need to add logic that simulates the CALL. Your program won't produce correct results of course but the objective of this exercise is to remove REXX from the mix to see what impact it has on performance. If REXX is your problem, then replace your REXX logic with COBOL logic.
Back to top
View user's profile Send private message
insomniaa

New User


Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes

PostPosted: Wed Aug 15, 2007 8:40 pm
Reply with quote

thanks TG...

I have already substituted the REXX call with the standard call.

The process is already running...

I have also amended a few things to avoid repetitive processing in the program... lets see if this improves the execution time....

Thanks for the comments on this!!!

Leo
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Wed Aug 15, 2007 11:17 pm
Reply with quote

Hello,

Just to add - while rexx is very powerful, it is also very cpu intensive. This is because it is interpreted rather than "executed".

If a "thing" needs to be done once in an online "hit of the enter key" or once at the start of some batch process, if is nearly invisable. When it is used many, many times as in your batch process, it will take many, many cpu cycles.
Back to top
View user's profile Send private message
insomniaa

New User


Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes

PostPosted: Thu Aug 16, 2007 1:36 pm
Reply with quote

OK, thanks a lot!!! We will have a meeting with our labs and we will discuss this issue... once again, thank you for your time and help!!!!

Leo
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Thu Aug 16, 2007 6:19 pm
Reply with quote

You're welcome icon_smile.gif

We're here when new questions come up.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> COBOL Programming

 


Similar Topics
Topic Forum Replies
No new posts Run rexx in batch job CLIST & REXX 7
No new posts Excuting store procedure via JCL batch JCL & VSAM 1
No new posts SORT on detail record, then repeat he... DFSORT/ICETOOL 3
No new posts batch SFTP job using AOPBATCH unable ... All Other Mainframe Topics 7
No new posts Error to invoke MPP program through B... IMS DB/DC 3
Search our Forums:

Back to Top