IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Tivoli job scheduler issue


IBM Mainframe Forums -> IBM Tools
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
John Poulakos

Active User


Joined: 13 Jun 2012
Posts: 178
Location: United States

PostPosted: Sat Mar 23, 2019 12:32 am
Reply with quote

Here's one everyone is scratching their heads over, even IBM:

1. We changed a Proc in our Proclib
2. The old version of the proc was moved to a backup library, just in case.
3. We added the changed proc to a job that has been scheduled for several months.
4. The job ran from the schedule but picked up the old version of the proc.

The jcl expansion clearly shows the proc was picked up from the prod proclib where it was supposed to be.

The job clearly shows it executing the proc, so it had to pick up the changed job JCL.

The job has never specified this proc name before.

So, how did it use the old proc?


Old JCL (excluding comments, etc:

//FS267D JOB (WG1158,FIS),ADM,CLASS=C,MSGCLASS=M
//FS253D01 EXEC FS267

NewJCL:

//FS267D JOB (WG1158,FIS),ADM,CLASS=C,MSGCLASS=M
//FS253D01 EXEC FS253

Proc FS253 is the proc that was changed.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales

PostPosted: Mon Mar 25, 2019 12:37 pm
Reply with quote

Was this a once off baffler, or does it continue each run ?

Maybe the timing was just unfortunate where OPC had already prepped the job ready to run after the PROCLIB change - Just a suggestion
Back to top
View user's profile Send private message
David Robinson

Active User


Joined: 21 Dec 2011
Posts: 199
Location: UK

PostPosted: Mon Mar 25, 2019 8:43 pm
Reply with quote

I've seen this before, but never got to the bottom of it. In our case the attempted exec of the proc was immediately after the change, so we assumed it must have been something in storage somewhere that had not been refreshed.

Would be interesting to hear what the cause was if you find it.
Back to top
View user's profile Send private message
John Poulakos

Active User


Joined: 13 Jun 2012
Posts: 178
Location: United States

PostPosted: Tue Mar 26, 2019 9:42 pm
Reply with quote

Problem appears to be JES2 is holding an old version of PROCLIB. TWS may be holding an enqueue, but no "smoking gun" found. IBM called and several attempts have been made to fix the issue by recycling various components and reinstalling TWS. This has caused other problems. Now getting IEF344IF and IGD17358I errors on GDGs that are used only in one job that has never failed and no reclaim should ever be needed.

We rewrote the JCL to eliminate the failing PROC expansion, so I don't know if the original problem has been fixed.
Back to top
View user's profile Send private message
steve-myers

Active Member


Joined: 30 Nov 2013
Posts: 917
Location: The Universe

PostPosted: Tue Mar 26, 2019 10:27 pm
Reply with quote

Mr. Poulakos analysis is the right general idea, but possibly not correct.

When a job is submitted to JES2, it is "converted" (usually) almost immediately. "Conversion" analyzes the JCL. Any procedures in the job are read and analyzed during "conversion." Any proc changes made between "conversion" and actual execution are not reflected in the execution environment as the JCL has already been processed.

Another possible problem is in multi-access SPOOL if it is being used. The job may be "converted" on a different physical system than the execution system, so a different physical proclib may be used for conversion than on the execution system, and this can result in problems if the production control analysts are not wary.
Back to top
View user's profile Send private message
John Poulakos

Active User


Joined: 13 Jun 2012
Posts: 178
Location: United States

PostPosted: Tue Mar 26, 2019 10:48 pm
Reply with quote

The current and highly speculative belief, is the person responsible for space management was working on expanding PROCLIB when the failing PROC update was made. Exactly what was being done to PROCLIB is unclear.

The IEF344IF and IGD17358I errors appear to be the result of somebody manually submitting a job that is normally controlled by TWS. TWS was taken out of service at the time. I can see the GDG was correct and OK in the previous run of the job and bad in the following run of the job. But, the JES log is missing for the job that was submitted manually and caused the GDG error.

I think this is all I will ever know about this event. I doubt anyone will be forthcoming with better information.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales

PostPosted: Wed Mar 27, 2019 12:34 pm
Reply with quote

I always believed that if a physical change, (e.g. Deleted and redefined or moved to a different volume) was made to a PROCLIB library that JES had to be bounced.

I recall many many years ago when my colleague moved a PROCLIB and all hell broke loose until JES was bounced.
Back to top
View user's profile Send private message
John Poulakos

Active User


Joined: 13 Jun 2012
Posts: 178
Location: United States

PostPosted: Thu Mar 28, 2019 7:01 pm
Reply with quote

I think that is exactly what happened, but I will never be able to prove it. My job is "fixer". I get all significant problems, identify the cause and do whatever is needed to prevent them from occurring again. It requires co-operation from all areas of IT, which isn't always forthcoming; especially when the problem is human error by tech support. So, sometimes I get the oats after they have been through the horse.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales

PostPosted: Fri Mar 29, 2019 12:34 pm
Reply with quote

Just as another possible add on ............. senior moments seem more frequent threse days icon_rolleyes.gif

You were saying about increasing the space for the PROCLIB, so it could also be possible that he has ran a silly little IEFBR14 against the PROCLIB to allow it secondary allocations, which unfortunately may be used by jobs to update the PROCLIB but will not be recognised by JES until it gets bounced.

I think that if you really did fancy trolling through SMF that you could prove it icon_biggrin.gif

It's amazing the little snippets of old that traverse the grey matter sitting in the pub.
Back to top
View user's profile Send private message
John Poulakos

Active User


Joined: 13 Jun 2012
Posts: 178
Location: United States

PostPosted: Sat Apr 06, 2019 1:45 am
Reply with quote

Bingo. That turned out to be exactly what happened. I had completely forgotten about allowing secondary extents. But, I have an excuse... I'm even older than you, I'll bet. I'm 75.
Back to top
View user's profile Send private message
steve-myers

Active Member


Joined: 30 Nov 2013
Posts: 917
Location: The Universe

PostPosted: Sat Apr 06, 2019 9:28 am
Reply with quote

If the new member is in the new extent, the converter will barf trying to read the member. .. BUT the TS claimed it got the old member .. so something else happened.

There are ways to reopen the proclib concatenation if you got this extent problem. The easiest is to have a handy dandy job that specifies /*JOBPARM PROC=xx, where XX is an identifier for a non standard PROC in the JES2 procedure. The job doesn't even have to use a proc in the library! The next job will reopen PROC00.
Back to top
View user's profile Send private message
steve-myers

Active Member


Joined: 30 Nov 2013
Posts: 917
Location: The Universe

PostPosted: Sat Apr 06, 2019 10:54 am
Reply with quote

John Poulakos wrote:
...Old JCL (excluding comments, etc:

//FS267D JOB (WG1158,FIS),ADM,CLASS=C,MSGCLASS=M
//FS253D01 EXEC FS267

NewJCL:

//FS267D JOB (WG1158,FIS),ADM,CLASS=C,MSGCLASS=M
//FS253D01 EXEC FS253

Proc FS253 is the proc that was changed.


Wait one. This does not seem to be a PROCLIB issue. TWS is submtting a different, and unexpected JOB. Now the question seems to be, where did TWS get this job from? I'll have to bow out of this thread as I know virtually nothing about how TWS works.
Back to top
View user's profile Send private message
John Poulakos

Active User


Joined: 13 Jun 2012
Posts: 178
Location: United States

PostPosted: Fri Apr 12, 2019 7:56 pm
Reply with quote

The underlying problem WAS the PROCLIB expansion. But it messed up TWS pretty bad. I did not get all of the details, but IBM had to provide a resolution and it took two days to get the scheduled jobs running cleanly again.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> IBM Tools

 


Similar Topics
Topic Forum Replies
No new posts SFTP Issue - destination file record ... All Other Mainframe Topics 2
No new posts OPCP scheduler questio IBM Tools 0
No new posts Issue after ISPF copy to Linklist Lib... TSO/ISPF 1
No new posts Facing ABM3 issue! CICS 3
No new posts Panvalet - 9 Character name - Issue c... CA Products 6
Search our Forums:

Back to Top