View previous topic :: View next topic
|
Author |
Message |
chetanambi
New User
Joined: 21 Jan 2012 Posts: 58 Location: India
|
|
|
|
Please see the below is LISTCAT o/p as well. File was not being used when taking LISTCAT output:
Code: |
IDCAMS SYSTEM SERVICES TIME: 05:20:32
LISTCAT -
ENTRIES (XXXXX.XX.XXXX.XXXXXXX) -
CLUSTER -
ALL
CLUSTER ------- XXXXX.XX.XXXX.XXXXXXX
IN-CAT --- CATALOG.VVCATPW
HISTORY
DATASET-OWNER-----(NULL) CREATION--------2012.195
RELEASE----------------2 EXPIRATION------0000.000
SMSDATA
STORAGECLASS ----OLCACHE MANAGEMENTCLASS----OLSTD
DATACLASS ---------OLSTD LBACKUP ---0000.000.0000
CA-RECLAIM---------(YES)
EATTR-------------(NULL)
BWO STATUS------00000000 BWO TIMESTAMP---00000 00:00:00.0
BWO---------------(NULL)
RLSDATA
LOG ----------------(NULL) RECOVERY REQUIRED --(NO) FRLOG ---------
VSAM QUIESCED -------(NO) RLS IN USE ---------(NO)
LOGSTREAMID-----------------------------(NULL)
RECOVERY TIMESTAMP LOCAL-----X'0000000000000000'
RECOVERY TIMESTAMP GMT-------X'0000000000000000'
PROTECTION-PSWD-----(NULL) RACF----------------(NO)
ASSOCIATIONS
DATA-----XXXXX.XX.XXXX.XXXXXXX.DATA
INDEX----XXXXX.XX.XXXX.XXXXXXX.INDEX
IDCAMS SYSTEM SERVICES TIME: 05:20:32
THE NUMBER OF ENTRIES PROCESSED WAS:
AIX -------------------0
ALIAS -----------------0
CLUSTER ---------------1
DATA ------------------0
GDG -------------------0
INDEX -----------------0
NONVSAM ---------------0
PAGESPACE -------------0
PATH ------------------0
SPACE -----------------0
USERCATALOG -----------0
TAPELIBRARY -----------0
TAPEVOLUME ------------0
TOTAL -----------------1
THE NUMBER OF PROTECTED ENTRIES SUPPRESSED WAS 0
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 0
|
|
|
Back to top |
|
|
PeterHolland
Global Moderator
Joined: 27 Oct 2009 Posts: 2481 Location: Netherlands, Amstelveen
|
|
|
|
chetanambi wrote: |
Peter,
Even VERIFY is throwing same duplicate error message. Please see below:-
Code: |
IDCAMS SYSTEM SERVICES TIME: 05:07:16
VERIFY DATASET(XXXXX.XX.XXXX.XXXXXXX)
IDC3302I ACTION ERROR ON XXXXX.XX.XXXX.XXXXXXX
IDC3351I ** VSAM I/O RETURN CODE IS 156 - RPLFDBWD = X'D708009C'
IDC3003I FUNCTION TERMINATED. CONDITION CODE IS 12
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 12
|
Please suggest me..
Thanks,
Chetan |
This is another error (see the RPLFDBWD) and is described in "VSAM Demystified" |
|
Back to top |
|
|
chetanambi
New User
Joined: 21 Jan 2012 Posts: 58 Location: India
|
|
|
|
Peter,
I have looked into "VSAM Demystified" and if you see my previous messages I am doing the same 4 steps they have mentioned to correct corrupted index. But we are getting same abend occurs after few days or weeks or months.
For recovering the index:
– Use the REPRO Command to copy just the data component of the KSDS
to a sequential data set. Specify the data component name (not the
Cluster Name) in the REPRO INFILE parameter.
– Sort the sequential data set by key.
– DELETE and reDEFINE the damaged Cluster.
– REPRO from the Sorted Sequential File to the newly defined Cluster.
Record Management will rebuild the Index Component.
But I have noticed one thing in step 1 that it is copying upto the record which is causing the issue. Thru LISTCAT i found that its having 211310 records. But step 1 is copying only 202317 (and still thows return code 156).
My question is VSAM KSDS should not have out of sequence or duplicate records. rite? Why is this happening? How to avoid this?
Thanks,
Chetan |
|
Back to top |
|
|
PeterHolland
Global Moderator
Joined: 27 Oct 2009 Posts: 2481 Location: Netherlands, Amstelveen
|
|
|
|
Quote: |
But we are getting same abend occurs after few days or weeks or months. |
Well there must be somewhere a program that is f...ing your KSDS. So i think that you have to check your KSDS after every time it is used. |
|
Back to top |
|
|
Bill Woodger
Moderator Emeritus
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
|
|
|
|
I think Peter is spot on.
I've never seen anything like that, but something is trashing your KSDS such that the index is obtaining records out-of-sequence, at least on serial processing.
This explains how REPRO to a QSAM file gets your message, athough I never realised you could get that message from a sequence error on the input dataset.
Obviously you have a serious problem which may involve loss of data. As Peter suggests, check the KSDS after each completed use to find which particular program/s might be to blame and then get into that/those and find out what is going on.
Anything with Assembler accessing the KSDS? Especially if doing anything "clever"?
Just for marking something off, have you tried IDCAMS with EXEC PGM=IDCAMS rather than the "wrapper"?
VERIFY seems to spot the error, so you could try that, although I think I'd go for the REPRO as you know for certain it catches "something".
Good luck. Let us know the resolution please, as it increases our knowledge and may help someone with a similar problem in the future.
If you have more questions, someone'll be here.
Get your support/techies onto this immediately, make sure your boss is aware, there could be very serious implications. Review fault-reporting system for things which might relate to missing data or wrong data. |
|
Back to top |
|
|
expat
Global Moderator
Joined: 14 Mar 2007 Posts: 8797 Location: Welsh Wales
|
|
|
|
Peter, good point.
I have just noticed SHR(3 3) on the cluster definition |
|
Back to top |
|
|
chetanambi
New User
Joined: 21 Jan 2012 Posts: 58 Location: India
|
|
|
|
I did try with EXEC PGM=IDCAMS but the result is same ..
Ya.. in this case both VERIFY and REPRO are catching if there is something wrong with the file. I am really getting duplicate and out-of-sequence errors while copying to flat file. There is no question about it. As Peter and Bill are suspecting there might be an issue while adding/updating records into this file. I found that only 2 CICS programs are adding/updating the file. Most importantly this file being used by 2 different CICS regions and that's the reason share options (3,3).
Anyway I am going to look into these 2 programs what it is doing. I will not give up until this is solved of course you guys will be there to help me.
Lastly can anybody please explain me this:- VSAM should be handling out-of-sequence and duplicate conditions when record is getting added even if SHR(3,3). Why its not happening?. or anything wrong with cluster definition or something else?
Thanks,
Chetan |
|
Back to top |
|
|
Bill Woodger
Moderator Emeritus
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
|
|
|
|
If you have two processes updating the same VSAM file at the same time you will make a mess of the file unless you use a seperaate method to control which process updates when.
If one has just read an index, the second reads the same index, the second inserts a record, the first inserts a record, then your index has been compromised, and possibly your data as well.
Without looking in detail at what particular sequence of events caused a particular problem, you won't be able to predict the outcome beyond how Peter has already described it. It's that one begining with the letter F. |
|
Back to top |
|
|
dbzTHEdinosauer
Global Moderator
Joined: 20 Oct 2006 Posts: 6966 Location: porcelain throne
|
|
|
|
this info may help you fix the dataset Broken Dataset
as far as the problem is concerned,
there is a little blurb in VSAM Dym... (page 153) that states:
Quote: |
Sharing a data set across regions (cross-region) or across systems
(cross-system) without using proper enqueuing procedures to protect data set
integrity. For shareoptions shr(3 3) shr(4 3) shr(3 4) see related information in
OY36328.
Sharing a data set across systems — even using the appropriated share
option — but without propagating ENQ name SYSVSAM around the GRS
ring. This is the *most* common user error. It results in duplicate index
pointers in the high level index records. See message IDC11705I |
|
|
Back to top |
|
|
Robert Sample
Global Moderator
Joined: 06 Jun 2008 Posts: 8696 Location: Dubuque, Iowa, USA
|
|
|
|
Quote: |
VSAM should be handling out-of-sequence and duplicate conditions when record is getting added even if SHR(3,3). |
Sorry, but you are making a false statement here. From the AMS for Catalogs manual on SHROPTIONS (with emphasis added by me):
Quote: |
3
Specifies that the data set can be fully shared. With this option, each user is responsible for maintaining both read and write integrity for the data that user's program accesses. User programs that ignore write integrity guidelines can cause VSAM program checks, uncorrectable data set errors, and other unpredictable results. This option requires each user to be responsible for maintenance. The RESERVE and DEQ macros are required with this option to maintain data set integrity. (For information on using RESERVE and DEQ, see z/OS MVS Programming: Authorized Assembler Services Reference ALE-DYN and z/OS MVS Programming: Authorized Assembler Services Reference LLA-SDU.) |
In other words, your site has caused this mess by not following IBM standards -- you're going to have to either change the SHROPTIONS or add enqueue / dequeue / reserve logic to the programs in each region to stop the problems -- or they will continue. |
|
Back to top |
|
|
vasanthkumarhb
Active User
Joined: 06 Sep 2007 Posts: 275 Location: Bang,iflex
|
|
|
|
Hi.
Quote: |
Vasanth,
Yes.. there are more steps in my job. This is what it doing:-
Step1: Copy VSAM fiel to flat file using REPRO.
Step2: Process the flat file and create o/p file out of it.
Step 3: Delete and Define the VSAM.
Step 4: Load data from o/p of step 2 to VSAM file using REPRO (with REPLACE option). |
By the by first tell me on which step you are getting sequence error.
Points below can encounter:-
1. Check whether you correctly sorting on key field also on mentioned length of the key..
2. If this error arriving on every week or 2 months. Try to have powerful key after discussing with your peers. Also verify your input shouldn't vary on these days.
3. Since two CICS region involved adding/updating on VSAM file. verify any user adding messy data. |
|
Back to top |
|
|
Robert Sample
Global Moderator
Joined: 06 Jun 2008 Posts: 8696 Location: Dubuque, Iowa, USA
|
|
|
|
Vasanth, the VSAM shareoptions are (3 3) -- unless all programs accessing the VSAM file in both CICS regions use enqueue / dequeue mechanisms, there is no need for "messy data" to have problems. Merely having both regions reading and writng to the VSAM file without using a queueing mechanism can cause the problems described.
The sort doesn't matter -- and could even be right at all times.
The user input doesn't matter -- and could even be right at all times.
The data quality doesn't matter -- and all data may be perfectly good at all times.
The only thing that matters is whether or not all programs accessing the VSAM file in both CICS regions use the proper way to ensure read and write integrity. If they don't, errors WILL occur. |
|
Back to top |
|
|
chetanambi
New User
Joined: 21 Jan 2012 Posts: 58 Location: India
|
|
|
|
After reading Bill's description and info provided by dbzTHEdinosauer and Robert, now I am clear about what is causing the issue. This is indeed an issue with improper use of share-options or not following IBM standards on using ENQ/DEQ/RESERVE logic in the programs.
As of now I am not aware these commands on how to use in my CICS application. So I am going thru "VSAM demystified" to get clear picture. Please let me know if you know any useful links for these.
I will reply as I get any progress. I am very happy that even though issue is not resolved I have already learnt many things on VSAM.
Thanks,
Chetan |
|
Back to top |
|
|
Robert Sample
Global Moderator
Joined: 06 Jun 2008 Posts: 8696 Location: Dubuque, Iowa, USA
|
|
|
|
In addition to VSAM Demystified, look at the CICS Application Programming Reference manual for the EXEC CICS ENQ and EXEC CICS DEQ commands. |
|
Back to top |
|
|
Bill O'Boyle
CICS Moderator
Joined: 14 Jan 2008 Posts: 2501 Location: Atlanta, Georgia, USA
|
|
|
|
Yup, after you get this fixed, it will happen again.
Alternatively, building/defining an FOR (File Owning Region) and defining this file in an FCT entry (along with others which are accessed similarly) would be beneficial in the long run.
If you go with the FOR, each current FCT entry in each given region (AOR), must specify the KEYLENGTH value, providing the program's file API doesn't, along with the SYSID of the FOR (REMOTESYSTEM). Without a KEYLENGTH in the API as well as being omitted from the FCT entry, an INVREQ will be raised. With that, it's much easier to specify the KEYLENGTH value along with the mandatory REMOTESYSTEM/SYSID value in the FCT entry.
FWIW, when restarting regions, the FOR is always the first one.... |
|
Back to top |
|
|
chetanambi
New User
Joined: 21 Jan 2012 Posts: 58 Location: India
|
|
|
|
Bill,
Even after changing the share-options or using those CICS commands to ensure read and write integrity, problem will occur again??. Is that what you want to say?
I am not very clear about other part of your reply. The VSAM file is already defined in both the CICS regions with KEYLENGTH and I have verfied this thru CEMT command. This is what I see there:-
Code: |
File(xxxxxxx)
Accessmethod(Vsam)
Openstatus( Open )
Enablestatus( Enabled )
Readstatus( Read )
Updatestatus( Update )
Addstatus( Addable )
Browsestatus( Browse )
Deletestatus( Delete )
Exclstatus( )
Disposition( Share )
Rlsaccess( Notrls )
Emptystatus( Noemptyreq )
Dsname( XXXXX.XX.XXXX.XXXXXXX )
Table( Nottable )
Loadtype( Noload )
Cfdtpool( )
Tablename( )
Updatemodel( Locking )
Maxnumrecs( 00000000 )
Keylength( 014 )
Recordsize( 00106 )
Rbatype(Notextended)
Remotesystem()
|
Thanks,
Chetan |
|
Back to top |
|
|
Bill Woodger
Moderator Emeritus
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
|
|
|
|
No, I'm not saying that.
Your shareoptions allows multiple concurrent opens-for-update.
If you change your shareoptions to allow only one open-for-update at a time, you won't get the problem (but won't have simultaneous access in two CICS regions).
If you do the enqueing process as outlined by people above then you can have the file open in multiple programs but you will have a method whereby only one update of any type (add/change/delete) can be done at any one time (presuming coded correctly). As only one update can be done at a time, there is no possibility of "losing" an chunk of the index through two or more updates being done at the same time.
Don't get worried about "performance". It is pretty zippy and it is the way everyone does it and it has to be done. |
|
Back to top |
|
|
dbzTHEdinosauer
Global Moderator
Joined: 20 Oct 2006 Posts: 6966 Location: porcelain throne
|
|
|
|
Bill O'Boyle wrote: |
Yup, after you get this fixed, it will happen again.
Alternatively, building/defining an FOR (File Owning Region) and defining this file in an FCT entry (along with others which are accessed similarly) would be beneficial in the long run.
|
hmmmm, I wonder why Bill brought this up?
probably because when you have an enq/deq from every request
within an Appllication Owning Region,
having two regions with implicit FOR (as your site is set-up)
is the same as having no enq/deq.
you can only effectively enq/deq a resource one place,
in this case an FOR.
hmmmm, wonder why IBM came up with the idea of AOR and FOR?
Hmmm???
can both regions have cause to access the same records?
if not, maybe divide the VSAM into two VSAMs based on the logic driving each region.
Then you don't have to have you storage management people learn about AOR and FOR. |
|
Back to top |
|
|
Anuj Dhawan
Superior Member
Joined: 22 Apr 2006 Posts: 6250 Location: Mumbai, India
|
|
|
|
This thread is a keeper - but, why not redefine VSAM with SHR(2 3)? If you want to do batch update processing on a VSAM file that is used by CICS, use CEMT to close it within CICS, so allowing a batch job to be the single permitted writer, no? |
|
Back to top |
|
|
chetanambi
New User
Joined: 21 Jan 2012 Posts: 58 Location: India
|
|
|
|
I have gone thru some of the links provided. For the devices shared between multiple z/OS systems, enqueuing is accomplished though RESERVE and DEQ. and if one system ENQ and DEQ can be used. I know about ENQ and DEQ commands but didn't find any info on RESERVE from the net. Please provide me if you have any information on RESERVE. or if I use ENQ or DEQ commands is that ok?
Thanks,
Chetan |
|
Back to top |
|
|
PeterHolland
Global Moderator
Joined: 27 Oct 2009 Posts: 2481 Location: Netherlands, Amstelveen
|
|
|
|
Reserve :
MVS Programming: Authorized Assembler Services Reference, Volume 3 (LLACOPY-SDUMPX) |
|
Back to top |
|
|
Bill O'Boyle
CICS Moderator
Joined: 14 Jan 2008 Posts: 2501 Location: Atlanta, Georgia, USA
|
|
|
|
If you have more than one region attempting to UPDATE and you're using ENQ/DEQ, they will only be known in the region issuing the UPDATE. In other words, REGIONA won't know about REGIONB's ENQ/DEQ's because they're not GLOBAL/SHAREABLE within an LPAR or two or more REGION's.
This is why an FOR (in the same LPAR) could centralise access and give you better file integrity. |
|
Back to top |
|
|
chetanambi
New User
Joined: 21 Jan 2012 Posts: 58 Location: India
|
|
|
|
Quote: |
This thread is a keeper - but, why not redefine VSAM with SHR(2 3)? If you want to do batch update processing on a VSAM file that is used by CICS, use CEMT to close it within CICS, so allowing a batch job to be the single permitted writer, no?
|
No we can't redefine file with SHR(2,3) because dataset should be fully shared among number of users (but the programs which are adding/updating the records in the file are not maintaing the read and write integrity and is causing the issue). Howewer we did try with SHR(2,3) sometime back and it worked for somedays but issue started occuring again after that.
Thanks,
Chetan |
|
Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
|
|
|
|
Hello,
Probably not what you are looking for, but maybe it is time to convert this to a database. . .
And we appear to have gone far away from the "out of sequence"
d |
|
Back to top |
|
|
chetanambi
New User
Joined: 21 Jan 2012 Posts: 58 Location: India
|
|
|
|
Quote: |
If you have more than one region attempting to UPDATE and you're using ENQ/DEQ, they will only be known in the region issuing the UPDATE. In other words, REGIONA won't know about REGIONB's ENQ/DEQ's because they're not GLOBAL/SHAREABLE within an LPAR or two or more REGION's |
Bill, Yes. more than one regions (in our case 2 regions) may try to update the data. Since more than one regions are involved here how to make it GLOBAL/SHAREABLE?. I found some information in VSAM demystified about GSR (Global Shared Resource) but no explanation how to use. Is that what you mean by GLOBAL/SHAREABLE?. Could you please explain me how can i use it for better file integrity?
So I am gonna try by changing share-options(2,3) once more time AND/OR with GLOBAL/SHAREABLE after I get information on it.
I have noticed that CEMT command for file is showing RLSACCESS as below. How about using the file in RLS (Record Level Sharing) mode across LPAR. Will that solve the problem?
Code: |
Rlsaccess( Notrls ) |
|
|
Back to top |
|
|
|