GDG gens missed when they are remotely created without break

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

I am facing a classic problem which has many classic solutions but none of them can be used because of the special constraints that I am supposed to work within. I could not find a solution that is guaranteed to work across system level upgrades.

So I am hoping that this august forum would bless me with a solution.

Our mainframe based application receives data from remote distributed systems via various file transfer protocols, NDM, FTPS, etc. These files arrive in quick succession, sometimes a few seconds within one another.

They are also accomapanied by TWS triggers that kick off the jobs that process them.

The files are written as generations under the same GDG. My jobs are designed to read the (0) gen of the files. So, if gen1 is written and the trigger is set and almost simultaneously gen2 is written and the trigger is set again, JobA gets triggered twice and 2 JobAs with different jobnums gets triggered and wait to run.

When the first JobA runs gen2 would already be cataloged. So gen1 does not get processed. I wrote a process using rexx to do a listcat of the gdg everytime and use the saved g0000v00 name from the prevous run to locate new generations of the file and thus sequentially step through the g0000v00s and prevent them from being missed.

This has been working more or less but has 2 problems. 1) on rare occassions TWS does not trigger jobs even if SRSTAT triggers are set. On those occassions, new g0000v00s get processed only during the next run. This introduces a lag in the processing. 2) This solution is not guaranteed to work across ystem upgrades since it lists g0000v00s using listcat and if the format of listcat output chalnges there is a remote possibilty that the process may not conitnue to work. I have enhanced the process to set alerts if there a re additional new generations. I have also made the process more independent of the format of listcat.

But I have been told to look for solutions that are guaranteed to work beyond armageddon. Is there any such soultion available ?

We cannot a) Ask the sending systems to put a 'delay' between 2 successive files b) Cannot hold the incoming g0000v00s and have them concatenated and processed periodically by scheduled processes c) Files have to be send out 'as and when they' arrive there is no question of 'holding them'. d) we have to use GDGs. no file names with date and time and d) i will be shot if I mention hfs files or java.

De-blocked

Nic Clouston · Posted: Sat Dec 14, 2013 4:45 pm

Why not have each transfer write to a specific, non-gdg, dataset and after the dataset has been processed copy it to a gdg? This is how it is generally done in my personal experience.

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

I don't think this would solve the issue. It could cause the data to be over written and completely lost. 1) remote system writes Batch1 to input sequential file and sets srstat trigger. 2) Jobnum1 is triggered by tws and it comes into queue but does not start running yet. 3) Remote system overwrites input seq file withBatch2 and sets trigger 4) jobnum2 comes into queue. 5) Jobnum1 gets initiator and starts running and processes batch2 from input seq file. It finishes 6) jobnum2 starts running, processes batch2 again and finishes

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

Batch1 is lost since it was over written. Jobnum1 and Jobnum2 were 2 instances of JobA that was trying to copy the input seq file to the gdg

Robert Sample · Posted: Sat Dec 14, 2013 6:51 pm

1. If the job doesn't get started, there is nothing you can do about processing lag, unless you start a job every so many minutes to see what work is out there to do. Your site needs to investigate what is causing TWS to not submit the job at the appropriate time.

2. Use dynamic allocation instead of reading LISTCAT output. Dynamically allocate successive generations and process them until you get a dynamic allocation failure.

PeterHolland · Posted: Sat Dec 14, 2013 7:59 pm

I would set a trigger BEFORE the data is received.

And jobs submitted by TWS and not able to run? Then there is something terrible wrong with the job scheduling/job class/initiator definitions.

I, at least, never happened to see that.

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

The scheduling glitch that results in the job not getting triggered occurs very rarely. I have been told that happens when an ETT is set at the time when the current days plan is loading. But that is only one of issues. I need something that has no chance of breaking as a result of a future system upgrade.
I am required to process the gens in the order they arrive so I cannot process (0) gen and work backwards. So I think I do need save the g0000v00 and do listcat. The new generation to be processed is written to sequential file, picked up by a subsequent step allocated dynamically and processed

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

Even if there was no scheduling glitch the issues of gens getting skipped would still remain because the jobs getting intiators and beginning to run is independent of new generation s being created by remote system. Ideally the remote system should wait for a signal from mainframe after a gen is processed before sending the next gen. But we cannot ask the remote system to change their process

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

If issue will be there even if the trigger is set before the file sent. It will guarantee that the first job would have run a completed before the next file is sent by remote system

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

Robert samples solution would have been good if I did not have to process the older generations first. I could have dynamically read each generation and moved them off the gdg one by one via the rexx

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

Is there a way to dynamically process generations staring with the oldest one without doing the listcat. Listcat is ine wilcard that is being used that could potentially produce output in a different format after a future system level upgrade.

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

Catalog search interface via rexx ? Would that be unaffected by future system changes?

gcicchet · Posted: Sun Dec 15, 2013 3:25 am

Hi,

even if you manage to correctly generate the job(s) with the correct GDG(s),
in the event of of 2 jobs waiting to run and the fisrt one fails, you will now process files out of sequence unless the process aborts with incorrect sequencing.

Gerry

Nic Clouston · Posted: Sun Dec 15, 2013 3:44 am

Why not implement a signalling system: a file is not sent to you until the sender receives confirmation from you that the last file was processed?

Robert Sample · Posted: Sun Dec 15, 2013 4:28 am

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

As I said before 1) the existing process does save the generation 2) it does use it as a pointer in listcat listing to find the next generation. 3) a subsequent step then dynamically alocates and processes the gen. All of that is already being done. And the process does work ordinarily. But please read this. The main issue is the reliance on listcat to identify the gens that are there. The listing is parsed based on certain assumption s about the format. If these assumptions become false due FUTURE SYSTEM UPGRADES the process may break. THAT is the main issue. If there is a way to process gens in Fifo order without using listcat we will have process that will work across system upgrades. Plea se let me know

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

See, if this had been possible. I would not have to save the generation or do a a listcat. I simply would have had to dynamically read each generation and move them off the gdg. So the next time the job reads the gdg the processed generation would not be there. So the next gen gets picked up. BUT I need to process fifo order. So this wont work.

Robert Sample · Posted: Sun Dec 15, 2013 7:18 am

You do NOT need to run LISTCAT to find generations -- if the dynamic allocation failed, look at the return code to see why.

And your process is broken if you think LISTCAT is required -- even more broken when you think a system upgrade will change LISTCAT to make your process not work. Why are you spending all this time worrying about something THAT MAY NEVER OCCUR? And even if LISTCAT output is changed, it would take minutes to change the code to fix the problem -- so you have ALREADY spent far more time and energy worrying about this "problem" than could ever be justified.

And do NOT try to say this is "the requirement" -- that is a phrase used on this forum by those who do not care to think (or cannot think) independently to solve system problems.

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

Do do see the wisdom in your words mr sample
I can use one of the dynamic allocation programs with rexx to pull dsns from dsns from (0) backwards till I find the saved dsn from last run and then pick the dsn directly after that. No need for listcat. Thank you so much for your patience and time. You rock

Nic Clouston · Posted: Sun Dec 15, 2013 11:42 pm

Or you could read the file of holding the last generation processed and then allocate the next - no need to read backwards through the gds's. Simply read forward processing one at a time until allocation fails due to no more generations left to process.

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

Are you suggesting to read next gen by incrementing g0000v00 number manually example: process g0002v00 if the saved gen is g0001v00, without using reading using relative numbers (0), (-1) and so on ?

Nic Clouston · Posted: Mon Dec 16, 2013 4:23 pm

Naturally. I do not know if you could dynamically allocate using relative generations so, of course, you have to specify the full DSN.

harisukumaran · New User Joined: 14 Jun 2005 Posts: 75

Manually incrementing the g0000v00 could lead to problems. Also I cannot start from the oldest gen and read forward without determining number of active gens. If you know how to do this without parsing a listcat listing, please let me know. I can use bpxwdyn to read gdg using relative numbers and get the DSN. If I start from (0) and read backwards I don't need to know in advance the numbers of active gens

expat · Posted: Mon Dec 16, 2013 5:43 pm

Why not have just one job that processes the GDG base, and then copies all data processed to a new GDG and delete the current input GDS datasets ?

Nic Clouston · Posted: Mon Dec 16, 2013 5:54 pm