Add a step counter in a stack ?

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Hi everyone, so this is my first post ever however I've been following this forum for so many years now and I always enjoy reading all the content.
So, HEY!

Basically, as I mentioned in the subject, I'm looking for a way to add a step counter inside my rexx stack, however, I reached a point where I'm not sure where to go from here.. basically, my rexx will read data from 2 datasets (source / target) and for each 1 row that is read on these datasets, 4 job steps are created, so first I do write a job card to the member and then once each dataset is read those from source/target the new 4 steps are created, so every time the stack resets and goes over again until it reaches the end of the file..

Is there a way to put a step counter for each step that it is generated, however, do not reset this counter ?? I was investigating MAKEBUFF command.. perhaps this is what I am looking for ??

Follow the code;

As you may see, each step has a STEP'S' on it, I tried putting a counter with S = 0 ; S = S + 1 but it didnt work due the stack I guess..

Appreciate the support if possible.

Thanks !

prino · Posted: Fri Jul 17, 2020 10:56 am

Two do's, one end. Your code will not work.

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Now that you mentioned... its weird but its working as I need.... will look into it thank you.. but still haven't figured out that counter thou.. :-(

Willy Jensen · Posted: Fri Jul 17, 2020 7:22 pm

Variables are not affected by the stack. Both DOs are using the same control variable, surely that is not intended? And I can't see you setting the S variable anywhere. You need to initialise S before the first DO and then update it inside the second DO.

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Thanks Willy for the tip, so I managed to add the step counter.
I know the code doesn't look perfect, I still need to go thru and review it and will spend some time always to improve and remove unnecessary stuff and adding more stuff, I always wanted to do coding so I am enjoying it each successful step (and learning with the bad choices lol)

So, following your tip, I managed to add the step counter.. but I got hit by another issue.. for each listed dataset in my input file, 4 steps are generated.. so the step counter is adding a value on a 4 step scale .. would there be a way to add one step counter for each one of the 4 steps ? like .. something continuous STEP1 , STEP2 , STEP3 , STEP4 , STEP 5 , STEP6.. perhaps create one variable for each of the 4 QUEUED steps ?

Here's the code now;

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Sorry for double post, just to provide an example on how the job gets created after running code updated provided in my last update;
So it will generate a STEP1 4 times , then moves to STEP2 4 times.. and so on .. what I'm trying to accomplish here is to setup some variables that for each step it will add a step counter and keep updating that counter once every dataset is referenced.

Nic Clouston · Posted: Sat Jul 18, 2020 1:46 am

Willy Jensen · Posted: Sat Jul 18, 2020 2:50 am

Updated sample. I have made the assumption that the stems DSNL and DSLT are paired, so that source dataset 1 is in DSNLS.1 and target dataset 1 is in DSNLT.1, and so forth. Hence you only need one loop. Then I understand that you want each and every step number increased. The step number will be right-adjusted 4 bytes wide left-padded with 0. Finally I have used EXECIO directly instead of calling the WRITE routine. I hope that this is what you had in mind.

Joerg.Findeisen · Posted: Sat Jul 18, 2020 3:16 am

A job can have a maximum of 255 job steps. This maximum includes all steps in any procedures the EXEC statements call.

Also remove the unnecessary SLEEP step and modify the HSEND to make use of the WAIT parameter instead.

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Hi Willy ! The sample you wrote worked exactly the way I needed !
Thank you very very much for your help, and indeed, these source/target dataset records works in pair, so whenever a record is added on those steps from source, the same record (however with different HLQ) will be added on TGTNAME.

yes, in fact I'm expecting thousand lines to be retrieved but wanted to add the step counter so I can split the jobs across different members whenever it reaches the maximum number of 255 steps (240 is the number I've had in mind).

Also, thanks for explaining each update you've made to the sample, the good thing is that I was able to understand so I believe I'm getting there.

Thanks again!

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Joerg.Findeisen · Posted: Sat Jul 18, 2020 10:57 am

@rodferrn: When you are able to issue DFHSM administrator commands, do not mix with user commands.

Change:

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Joerg, I really have no words to thank you enough for all the good suggestions.

By removing the SLEEP step (it also reduced A LOT the number of steps in the jobs) and adding the HSEND WAIT command on both FRRECOV and MIGRATE it reduced the jobs runtime considerable, below, same number of steps and same datasets ;

Old run using sleep;
2.38 MINUTES EXECUTION TIME

New run without Sleep and adding WAIT parm
0.20 MINUTES EXECUTION TIME

Thank you folks!! Will keep trying my best !

Willy Jensen · Posted: Sat Jul 18, 2020 5:04 pm

Well, if you really want to reduce the number of steps, then remember that the ALTER command can also be issued from TSO. So you could run everything in one step. Or even from your REXX. Or write a small driver REXX doing one block of HSEND, ALTER and HMIGRATE and then your original REXX would just generate the call to that REXX. Many ways to skin that particular cat.

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Thanks for your feedback Willy, the problem I have in running everything in a single step is that, when it comes to multiple files (such as application vsam tablespace files) then I would need basically to double the amount of storage in my storage group, because first I would do the FRRECOV with newname (for all datasets) and then proceed with alter storage class and hsm move. So in order to have this workaround working I would need to treat each dataset at once after proceeding to the next one.

If you allow me to do a one more quick question, I'm trying to use the LMDINIT / LMDLIST for VSAM Cluster/Data however, even by coding the DSNDBC (to list only cluster datasets) still my output shows both cluster and data files, any ideas why ?

Follow the code;

Willy Jensen · Posted: Wed Jul 22, 2020 2:02 pm

Interesting, LMDLIST returns the data- and index components, even though they do not match the LEVEL parameter.
You can add STATS(YES) to LMDLIST and then test the ZDLSIZE variable for being null, meaning cluster.

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Hi Willy, thanks for you reply again, I've been reading about the STATS(YES) and ZDLSIZE but I'm still trying to make sense of those 2 lol.

Ugh, it is funny how we move from a plan to another when we are trying to design a solution .. I've just removed the N_Split variable because for that one I was spamming multiple jobs based on the number of rows, but once JCL has a 255 steps/execs constraint I have to limit jobs to 255, given the number of vsam cluster datasets I have for my application this might even generate hundreds of jobs.. so I'm looking into some way to run a single step where it invoke the rexx and the rexx script will run the 3 set of commands for all datasets ... right now I was trying to limit the jobs generation by step counter .. I tried IF S = 255 then call Write however this will limit one single job with 255 steps ..

Will look into something like you said (running a single JCL where the rexx is invoked and all commands are processed by the rexx itself).

Anyway, just with the good tips and suggestions this is how the procedure look so far ... (its small work but its decent work ha!)

Pedro · Posted: Fri Jul 24, 2020 4:21 am

I do not like that variable DSN is used for both of these DO clauses. I think it should be two different variable names (though I did not examine the earlier logic thoroughly.

Willy Jensen · Posted: Fri Jul 24, 2020 7:07 pm

The double 'DO's have been noted before. Something else, if you just want to look at catalog info then you can use the IGGCSI00 program, which does not incur the cost of reading the VTOC like LMDLIST STATS(YES) does. IBM supplies a sample in SYS1.SAMPLIB(IGGCSIRX) , but there are numerous other implementations. Search this forum, or have a look at my CSIREXX here

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Hello Pedro and Willy, wow!! Thank you so much for all the input.
Using Adress TSO just resolved all my problems with step limit in JCL.

The more I work on this project the more I feel like learning and eager to improve the code. So, I've generated an input member (SRCSAP) containing 45K datasets of SAPsid.DSNDBC.** , running a single job with this exec (using address TSO) is working perfectly, but the first runtime took 4 hours to complete (not bad, definitely not bad) but looking to improve the code or the efficiency of the solution, I removed the WAIT parameter (as suggested by Pedro) and was able to reduce the runtime to 3 hours and a half.
Now, on a second test I compiled my rexx and I managed reduce that runtime even more.

Now, I am thinking about serializing my execution (still using address tso) but instead of processing a single source/target member I was thinking about doing a dataset count by 10k or 12k and after each 10-12k generate a new member in DB2.CLONE.NEW.sid;

The current solution (below) will generate one source and one target member and add all source datasets (45k) on the same member, so I am looking for a way to do (perhaps a NESTED LOOP ? ) to generate a new member lets say SRCSPx and TGTSPx once 10 or 12k files are read..

Would a nested loop resolve this ??

This is the current code;

Willy Jensen · Posted: Wed Jul 29, 2020 1:02 pm

Something like (not tested, from the top of my head).
Notes:
1 I use BPXWDYN for allocation so that I can retrieve a generated DDname.
2. I use count= 1 to 999999 to save a coount-up operation. not a big saving but still.
3. When count is divisable with 10000 then a reallocation is done
4. I leave the loop for LMDLIST rc<>0, which I find simpler.
I hope this is somewhat what yo are loooking for.

Willy Jensen · Posted: Wed Jul 29, 2020 4:42 pm

Just realized that, despite my general preference for BPXWDYN over ALLOCATE, in this case "ALLOCATE DA('dataset.name(mbr"mbrn")) DD(ddname) SHR REUSE" is the simpler option.

rodferrn · New User Joined: 14 Jul 2020 Posts: 14 Location: Brazil

Hey Willy, thanks again for your reply, but I'm still having issues to write down the logic to split my DSNDBD files into multiple members.

Whenever you guys give me a sample, what I try to do is, instead of simply copy and paste I try to write down my own logic based on the provided code, in that way, plus reading manuals is the way I'm using to learn rexx (and programming actually).

Now, allow me to paste the entire logic below and explain a little bit (with the current logic I have 2 issues that I'm trying to adress).

So the logic basically asks for a SID (subsystem id) based on that SID, a dataset will be created and then LMDINIT / LMDLIST comes into play to list all datasets based on SAPsid.DSNDBD.** , now, using one database as example, I have 20165 DSNDBD files, so the logic has mainly 2 issues

1) Code is generating 2 members in DB2.CLONE.NEW.sid where SRCSAP1 goes from dataset number 1 to number 10.000 (which is ok) however the second member generated SRCSAP2 will copy the same datasets from row 1 to row 10.000 and continue until it reaches 20.002 (and then it stops). Now the second member is still adding the first 10k files (which were supposed to be on member 1 only).

2) Since its doing "IF count // 10001=0 THEN CALL Realloc" count goes only until 20.002 files, so its not writing all 20.165 files into the members.

Now, I have a guess about issue number 1, since I'm doing LMDINIT / LMDLIST, I think between the loop (from SRCSAP1 to SRCSAP2) the list is deleted and recreated so it will add all the 20002 datasets in member 2 ? I thought about creating a 3rd member containing the full list and then split this list into 2 new members splitting by 10.000 each ? Not sure if that would be the correct approach to follow ?

Here's the current code;

Nic Clouston · Posted: Fri Jul 31, 2020 3:16 pm

After writing your 10,000 records you then have to remove them i.e. DROP DATASAP - as you were previously told.