How to make REXX process different GDGs?

venksiv · New User Joined: 20 Jun 2015 Posts: 26 Location: INDIA

Hi all,

I am back requesting your inputs to resolve the below scenario:
* I am executing REXX exec using an event-triggered job.
* This job gets triggered via OPC scheduler whenever a new generation of GDG is created (through FTP/NDM).
* The exec is logically coded to read and process the latest generation
* Now, this logic holds good if there is a single event
* When multiple generations are created consecutively, multiple jobs will be in execution queue and since the job names are same (and unchangeable), there is a possibility of the first job to be consuming the latest generation
* However, this should not happen. No generation should be missed, no generation should be processed more than once and each job must process the corresponding file in order for the preceding conditions to be followed
* Furthermore, during the business hours, even though the jobs are getting triggered, they are not executing immediately due to operations restrictions in Dev region. This adds to the complexity of the scenario - the very first instance of the job will end up processing the latest generation.

Please share your expertise to achieve the desired processing logic - each job should process only one generation and all the generations must be processed.

Akatsukami · Posted: Thu Sep 24, 2015 8:03 pm

Have the exec write the absolute generation of the data set it processed. Have it select for processing the most recent generation not written out.

Pedro · Posted: Thu Sep 24, 2015 11:27 pm

enrico-sorichetti · Posted: Fri Sep 25, 2015 12:47 am

well ... not too much after all

suppose the ts writes two rexx scripts ...

the wrapper ...

lists ( IGGCSI00) the GDG involved

reads - DISP(OLD) a PS dataset with just one record with the last generation processed

loop forever
determines the generation to be processed ( leaves loop if no more )
calls the script that processes the <current> generation
on success rewrites the tracking ps
end of loop

expat · Posted: Fri Sep 25, 2015 10:21 am

Or maybe the job processes the whole GDG group, and deletes/renames the whole GDG group at the end

That's the way I've done it in the past

Pete Wilson · Posted: Fri Sep 25, 2015 10:30 pm

yes, or MOD to an existing flat-file and sweep that up at the end and write an EOF or reallocate it to empty it ready for the next cycle

venksiv · New User Joined: 20 Jun 2015 Posts: 26 Location: INDIA

Thank you very much for your inputs.

I am on long weekend and don't have connectivity to work from home. I will take your design inputs and work on the solution on Monday (IST).

Thanks, again.

daveporcelan · Posted: Mon Sep 28, 2015 5:49 pm

I second Expat's approach.

It has worked for me many times.

One addition, before processing the whole GDG group, create an empty +1 generation.

If there are no generations to process, you will get a JCL error.

venksiv · New User Joined: 20 Jun 2015 Posts: 26 Location: INDIA

Thanks, daveporcelan.

Some other task pre-empted and I had to hold the coding of this logic in my tool. But I am supposed to complete this as early as possible.

I will share my findings once I resume the coding.

Thanks, again.

venksiv · New User Joined: 20 Jun 2015 Posts: 26 Location: INDIA

Hi all,

I have coded the below logic and tested it manually. I also intend to test it during US business hours with actual system triggers.

Note: I allocated one Tracker file - this tracker will contain the absolute generation # that was processed during the previous execution. Hence, I will update this during every execution with the absolute generation # being processed.

Pseudo code:
1. Once the job is triggered, find the latest generation in DASD (using BPXWDYN or LISTCAT). Parse the absolute generation number (say, GEN1).
2. Read the Tracker file. Parse the absolute generation number (say, GEN2).
3. Compare the GEN1 and GEN2:
3.a) If GEN2 < GEN1:
Increment GEN2, parse the fully qualified GDG name and update the Tracker file in OLD disp. Free the Tracker file.
3.b) Else (i.e., logically GEN2 = or > GEN1)
This will indicate that something has gone wrong. Send an email to support team to verify the integrity of Tracker file and last generation that was processed.
4. Process the GDG corresponding to GEN2 (new value) for the current execution of the job.

I sincerely thank everyone who shared design perspective. In case I face any logical issues during the testing, I will share that too.

Nic Clouston · Posted: Thu Oct 08, 2015 5:48 pm

You should not update/free the tracker file until you have successfully processed the data.

JPVRoff · Posted: Tue Oct 13, 2015 8:43 am

I have a process like this. Each job simply processes and deletes the current generation. If I needed to, I'd create a backup of the file I was processing - but I don't need to.
Seems to work OK for me...

venksiv · New User Joined: 20 Jun 2015 Posts: 26 Location: INDIA

Nic,

Thanks for your valuable input. You are absolutely right about not updating/freeing the tracker file until the processing is done. However, a restart mechanism can't be built in place to process the failed request again (the job runs as event-triggered). The process will intimate the team about success or failure of processing of a file. If the process fails even before the email is sent, I will have to wait for the requester to come back for further analysis.

Since all the jobs are going to run sequentially (job names are same), I am going to depend on the tracker for identifying next file to be processed. Also, I can have another tracker-like file which maintains the request number, the gdg generation and the result (pass/fail) with the result being updated after the processing is done. This way I could know if the processing was done on a particular request no matter what the result was.

JPVRoff,

Thanks for sharing your logic. I believe your logic is suitable when there is one generation created at a time. The scenario being discussed involves presence of multiple unprocessed generations of request file when the job executes. This execution should process the oldest of the unprocessed generations and not the latest thought that will be the one referred to if (0) is used on DASD.

Thanks all for sharing your insights and helping me with the solution. My process is yet to be rolled out to users. Will share the issues after the implementation.

Pete Wilson · Posted: Sat Oct 31, 2015 4:56 pm

'Issues' should be resolved in Testing!

Good luck anyway.

venksiv · New User Joined: 20 Jun 2015 Posts: 26 Location: INDIA

Affirmative, Pete!

I wrote that with the intention of sharing real-time scenarios.

Scope is ever widening!

venksiv · New User Joined: 20 Jun 2015 Posts: 26 Location: INDIA

Hi all,

My tool is live now and it was able to process different generations of GDG sequentially using the tracker logic. Thank you for all your guidance on this.