|
View previous topic :: View next topic
|
| Author |
Message |
insomniaa
New User
Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes
|
|
|
|
Good afternoon all!!!
We are running a process that is having serious issues with performance and volume of information generated and wanted some feedback from you experts in batch processes:
1. How can repetitive call affect the performance of a batch program. Our process will at least make 2 million calls to the same cobol routine.
2. Our process currently generates a file with more than 130 million records that a filtered in the next step of the process (confrontation of two files). Would it be better to do everything in the same program bering in mind that the filter file has no more than 200 records. I mean, is it better a search all than a file confrontation?
Thanks!!!!
Leo |
|
| Back to top |
|
 |
William Thompson
Global Moderator
Joined: 18 Nov 2006
Posts: 3156
Location: Tucson AZ
|
|
|
|
Not quite sure of what you are asking, but...
Best would have the big file and the little file in the same matching key sequence - time spent would be almost all just reading and processing the selected data and very little matching.
Second best (if the big file can't be sorted) would be having the little file as binary searchable in-core table.
At 130 million repititions, even the adding of an increment to a ZD counter changed to a PD counter would help. And I'd sure look for others such as that.....Unsigned variables, even digit PD, extra subscript/index calculations...... |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
Please define "serious performance issues".
Searching in the first program while passing the 130mil records should be much faster than making another pass of the 130mil records.
If the 200 values were sorted before being loaded into an array, you could use a "SEARCH ALL" (binary search) and run even more efficiently. If you build a 200 entry array regardless of sequential or binary search, it is good practice to look at the "current" value and compare it to the "previous" value and if they are equal, do not search the array again as the entry in the array is already located.
Make sure the called module is reusable and not reloaded 2mil times. |
|
| Back to top |
|
|
insomniaa
New User
Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes
|
|
|
|
Thanks for the answer but not quite what I was looking for.
Our serious issues means that we have 10 JCL's doing the same logic (sweep process of customer database against a blacklist):
1. Each of the JCL's are or will be creating a file with at least 130 million records.
2.The process seems that it will be running for more than 5 hours.... never tested in our Preproduction env. but in Development it had ran for 17 hours and never finished because I cancelled it before.
The batch program performs a call to a cobol routine per read customer. If we have 2 million customers in the input file it will perform at least 2 million calls. Is this bad for the program performance?
In order to reduce the volume we thought about moving the filtering process into the same batch program, loading the filtering file into the working storage and perform a search all everytime is needed. Is this better, from a performance point of view to perform the filtering process in a different batch program?
I hope that this time is more understandable my enquiry... Cheer!!! |
|
| Back to top |
|
|
insomniaa
New User
Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes
|
|
|
|
Ummm.... so if I move the filter process to the same batch program I will reduce time and volume but I still need to reduce even more the execution time of the first batch program (the one that performs the sweep and call 2 million times to the same routine).
I am not sure if the routine is reusable, can i check this anywhere? our system is very limited and we don't have much information, my guess is that it is reusable but it is only a guess.
Once again, thank you for your help and time!!!! |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
You're welcome 
| Quote: |
| Is this better, from a performance point of view to perform the filtering process in a different batch program? |
I believe that if you load the 200 entry array and search it in the first set of code, you will perform better than if the process requires another step to pass all of the records again.
Is the module invoked via a static call or a dynamic call? If you use a static call and specify NODYNAM, your process should re-invoke the module without any interaction with the load ibrary. You need to ensure that the code initializes any working variables as they will not be reset on calls 2 thru n. |
|
| Back to top |
|
|
insomniaa
New User
Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes
|
|
|
|
Hi, thanks 
The module is invoked via a REXX function and it seems to be a dynamic call. In other words, the batch program invokes a rexx function to call the module.
Will the performance be better if we the code within the module is embedded directly into the batch program? In other words, we get rid of the module.
Leo
 Muchas gracias!!! |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
De Nada,
What service does the rexx "in the middle" provide?
If i could, i would have the "first" program call the sub-program directly and leave out the rexx in the middle. |
|
| Back to top |
|
|
insomniaa
New User
Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes
|
|
|
|
Dont know what the REXX is doing, i have tried to understand it but since I dont have experiece with REXX programming... but it seems that is the REXX the one invoking the module....
It is possible to make the first program to call the sub-program directly. Would this improve the execution time?
I can copy you the REXX function if you want....
Thanks!!! |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
| Quote: |
It is possible to make the first program to call the sub-program directly. Would this improve the execution time?
I can copy you the REXX function if you want.... |
I suspect it would improve the execution time if the main code directly called the sub-program - much depends on the "value added" by invoking the rexx code. How does the COBOL code invoke this rexx? Posting the COBOL "setup" for the call and the actual call may help us understand.
Sure, if you post the rexx code (please use the Code tag at the top of the replay panel for both the COBOL and rexx code), we can take a look at it. |
|
| Back to top |
|
|
dbzTHEdinosauer
Global Moderator
Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne
|
|
|
|
| If a dynamically called module does not contain the INITIAL statement, and it is not CANCELed, subsequent calls will find working-storage in a last used state. |
|
| Back to top |
|
|
TG Murphy
Active User
Joined: 23 Mar 2007
Posts: 148
Location: Ottawa Canada
|
|
|
|
My bet is that Rexx is your problem. As Dick suggested, show us the Rexx code so we can see whether the Rexx program is performing a crucial function - and if it is - whether this function can be carried out from within your COBOL program.
I don't think your COBOL sub-program by itself is your problem - provided you have made it serially reusable using the old fashioned way. The old fashioned way is to only use VALUE statements for defining constants - all variables should be initialized using MOVE, SET, etc..
As DBZ mentioned, you can make your program serially reusable using INITIAL or CANCEL but that's not the correct way to do it - you will suffer a performance hit using INITIAL and CANCEL. |
|
| Back to top |
|
|
insomniaa
New User
Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes
|
|
|
|
Hi all... many thanks for your comments. Our process has been running for 19 hours and going.... and we are talking of only one job when we actually have 10 jobs doing the same but with differents customers partitions....
This is the REXX function.
| Code: |
/* REXX */
ARG FX1VEZ FXNAME FXNUM LINE
IF LINE = '@MACRO@' THEN DO
QUEUE " EXEC-FUN "FXNAME" W-RUTINA USING W-PARAM1 "
QUEUE " W-PARAM2 "
QUEUE " ........ "
QUEUE " W-PARAMN "
QUEUE " END-FUN "
QUEUE "* Permite la llamada a rutinas desde programas Mixtos, "
QUEUE "* eliminado areas DFHEIBLK y DFHCOMMAREA para el proceso BATCH"
QUEUE "* y dejando los parametros tal y como los recibe para el "
QUEUE "* proceso CICS. "
QUEUE "* "
QUEUE "* W-RUTINA : Rutina a llamar. "
QUEUE "* W-PARAM1..N : Parametros de la rutina. "
RETURN 0
END
ELSE DO
PULL PARM
PULL PARM
RUTINA=WORD(LINE,2)
VUSING=''
IF SUBSTR(WORD(LINE,3),1,5)¬='USING' THEN DO
QUEUE "E* FALTA USING "; EXIT 8
END
QUEUE "P CALL "RUTINA" USING "
DO I=4 TO WORDS(LINE)-1
IF POS('CICS',PARM) = 0 THEN DO
IF WORD(LINE,I) = 'DFHEIBLK' |,
WORD(LINE,I) = 'DFHCOMMAREA' THEN ITERATE
END
QUEUE "P "WORD(LINE,I)
END
END
RETURN 0
|
And this is how the main program invokes the REXX function:
| Code: |
INITIALIZE BL25-SALIDA
REPLACING ALPHANUMERIC DATA BY SPACES.
MOVE ENT1-FIRST-NM TO BL25-FIRST-1
MOVE ENT1-MIDDL-NM TO BL25-MIDDLE-1
MOVE ENT1-LAST-NM TO BL25-SURNAME-1
MOVE ENT1-PAISRESI TO BL25-PAIS-RESI-1
MOVE ENT1-PAISNAC TO BL25-PAIS-NACI-1
MOVE ENT1-CPAIS2NA TO BL25-PAIS-2NAC-1
MOVE ENT1-FECHNACI TO BL25-FECH-NACI-1
EXEC-FUN
XX_CALL CA-BHCBL25 USING DFHEIBLK
DFHCOMMAREA
BL25-PARAMETROS
END-FUN
|
Hope this helps...
Thanks again!!!
Leo |
|
| Back to top |
|
|
TG Murphy
Active User
Joined: 23 Mar 2007
Posts: 148
Location: Ottawa Canada
|
|
|
|
Looks to me like Rexx was used so as to take advantage of its great text parsing abilities. In most cases - a reasonable thing to do.
However, the parsing carried out by the Rexx program could be done using COBOL code - although not nearly as easily. I would suggest this:
Start by modifying your program so that it does NOT call the REXX program. You may need to add logic that simulates the CALL. Your program won't produce correct results of course but the objective of this exercise is to remove REXX from the mix to see what impact it has on performance. If REXX is your problem, then replace your REXX logic with COBOL logic. |
|
| Back to top |
|
|
insomniaa
New User
Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes
|
|
|
|
thanks TG...
I have already substituted the REXX call with the standard call.
The process is already running...
I have also amended a few things to avoid repetitive processing in the program... lets see if this improves the execution time....
Thanks for the comments on this!!!
Leo |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
Just to add - while rexx is very powerful, it is also very cpu intensive. This is because it is interpreted rather than "executed".
If a "thing" needs to be done once in an online "hit of the enter key" or once at the start of some batch process, if is nearly invisable. When it is used many, many times as in your batch process, it will take many, many cpu cycles. |
|
| Back to top |
|
|
insomniaa
New User
Joined: 07 Mar 2007
Posts: 11
Location: Milton Keynes
|
|
|
|
OK, thanks a lot!!! We will have a meeting with our labs and we will discuss this issue... once again, thank you for your time and help!!!!
Leo |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
You're welcome
We're here when new questions come up. |
|
| Back to top |
|
|
|
|
