IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

RECORD OUT OF SEQUENCE error on VSAM KSDS file..


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
chetanambi

New User


Joined: 21 Jan 2012
Posts: 58
Location: India

PostPosted: Tue Jul 17, 2012 2:55 pm
Reply with quote

Please see the below is LISTCAT o/p as well. File was not being used when taking LISTCAT output:
Code:

IDCAMS  SYSTEM SERVICES                                           TIME: 05:20:32
                                                                               
 LISTCAT -                                                                     
  ENTRIES (XXXXX.XX.XXXX.XXXXXXX) -                                           
  CLUSTER -                                                                     
  ALL                                                                           
CLUSTER ------- XXXXX.XX.XXXX.XXXXXXX                                         
     IN-CAT --- CATALOG.VVCATPW                                                 
     HISTORY                                                                   
       DATASET-OWNER-----(NULL)     CREATION--------2012.195                   
       RELEASE----------------2     EXPIRATION------0000.000                   
     SMSDATA                                                                   
       STORAGECLASS ----OLCACHE     MANAGEMENTCLASS----OLSTD                   
       DATACLASS ---------OLSTD     LBACKUP ---0000.000.0000                   
       CA-RECLAIM---------(YES)                                                 
       EATTR-------------(NULL)                                                 
       BWO STATUS------00000000     BWO TIMESTAMP---00000 00:00:00.0           
       BWO---------------(NULL)                                                 
     RLSDATA                                                                   
       LOG ----------------(NULL)   RECOVERY REQUIRED --(NO)     FRLOG ---------
       VSAM QUIESCED -------(NO)    RLS IN USE ---------(NO)                   
       LOGSTREAMID-----------------------------(NULL)                           
       RECOVERY TIMESTAMP LOCAL-----X'0000000000000000'                         
       RECOVERY TIMESTAMP GMT-------X'0000000000000000'                         
     PROTECTION-PSWD-----(NULL)     RACF----------------(NO)                   
     ASSOCIATIONS                                                               
       DATA-----XXXXX.XX.XXXX.XXXXXXX.DATA                                   
       INDEX----XXXXX.XX.XXXX.XXXXXXX.INDEX                                   
IDCAMS  SYSTEM SERVICES                                           TIME: 05:20:32
         THE NUMBER OF ENTRIES PROCESSED WAS:                                   
                   AIX -------------------0                                     
                   ALIAS -----------------0                                     
                   CLUSTER ---------------1                                     
                   DATA ------------------0                                     
                   GDG -------------------0                                     
                   INDEX -----------------0                                     
                   NONVSAM ---------------0                                     
                   PAGESPACE -------------0                                     
                   PATH ------------------0                                     
                   SPACE -----------------0                                     
                   USERCATALOG -----------0                                     
                   TAPELIBRARY -----------0                                     
                   TAPEVOLUME ------------0                                     
                   TOTAL -----------------1                                     
         THE NUMBER OF PROTECTED ENTRIES SUPPRESSED WAS 0                       
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0                       
                                                                               
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 0                 
Back to top
View user's profile Send private message
PeterHolland

Global Moderator


Joined: 27 Oct 2009
Posts: 2481
Location: Netherlands, Amstelveen

PostPosted: Tue Jul 17, 2012 3:01 pm
Reply with quote

chetanambi wrote:
Peter,

Even VERIFY is throwing same duplicate error message. Please see below:-
Code:

IDCAMS  SYSTEM SERVICES                                           TIME: 05:07:16
                                                                               
 VERIFY DATASET(XXXXX.XX.XXXX.XXXXXXX)                                       
IDC3302I  ACTION ERROR ON XXXXX.XX.XXXX.XXXXXXX                               
IDC3351I ** VSAM I/O RETURN CODE IS 156 - RPLFDBWD = X'D708009C'               
IDC3003I FUNCTION TERMINATED. CONDITION CODE IS 12                             
                                                                               
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 12             

Please suggest me..

Thanks,
Chetan


This is another error (see the RPLFDBWD) and is described in "VSAM Demystified"
Back to top
View user's profile Send private message
chetanambi

New User


Joined: 21 Jan 2012
Posts: 58
Location: India

PostPosted: Tue Jul 17, 2012 3:54 pm
Reply with quote

Peter,

I have looked into "VSAM Demystified" and if you see my previous messages I am doing the same 4 steps they have mentioned to correct corrupted index. But we are getting same abend occurs after few days or weeks or months.
 For recovering the index:
– Use the REPRO Command to copy just the data component of the KSDS
to a sequential data set. Specify the data component name (not the
Cluster Name) in the REPRO INFILE parameter.
– Sort the sequential data set by key.
– DELETE and reDEFINE the damaged Cluster.
– REPRO from the Sorted Sequential File to the newly defined Cluster.
Record Management will rebuild the Index Component.

But I have noticed one thing in step 1 that it is copying upto the record which is causing the issue. Thru LISTCAT i found that its having 211310 records. But step 1 is copying only 202317 (and still thows return code 156).

My question is VSAM KSDS should not have out of sequence or duplicate records. rite? Why is this happening? How to avoid this?

Thanks,
Chetan
Back to top
View user's profile Send private message
PeterHolland

Global Moderator


Joined: 27 Oct 2009
Posts: 2481
Location: Netherlands, Amstelveen

PostPosted: Tue Jul 17, 2012 4:03 pm
Reply with quote

Quote:
But we are getting same abend occurs after few days or weeks or months.


Well there must be somewhere a program that is f...ing your KSDS. So i think that you have to check your KSDS after every time it is used.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Tue Jul 17, 2012 4:36 pm
Reply with quote

I think Peter is spot on.

I've never seen anything like that, but something is trashing your KSDS such that the index is obtaining records out-of-sequence, at least on serial processing.

This explains how REPRO to a QSAM file gets your message, athough I never realised you could get that message from a sequence error on the input dataset.

Obviously you have a serious problem which may involve loss of data. As Peter suggests, check the KSDS after each completed use to find which particular program/s might be to blame and then get into that/those and find out what is going on.

Anything with Assembler accessing the KSDS? Especially if doing anything "clever"?

Just for marking something off, have you tried IDCAMS with EXEC PGM=IDCAMS rather than the "wrapper"?

VERIFY seems to spot the error, so you could try that, although I think I'd go for the REPRO as you know for certain it catches "something".

Good luck. Let us know the resolution please, as it increases our knowledge and may help someone with a similar problem in the future.

If you have more questions, someone'll be here.

Get your support/techies onto this immediately, make sure your boss is aware, there could be very serious implications. Review fault-reporting system for things which might relate to missing data or wrong data.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales

PostPosted: Tue Jul 17, 2012 4:56 pm
Reply with quote

Peter, good point.

I have just noticed SHR(3 3) on the cluster definition
Back to top
View user's profile Send private message
chetanambi

New User


Joined: 21 Jan 2012
Posts: 58
Location: India

PostPosted: Tue Jul 17, 2012 7:56 pm
Reply with quote

I did try with EXEC PGM=IDCAMS but the result is same icon_sad.gif..

Ya.. in this case both VERIFY and REPRO are catching if there is something wrong with the file. I am really getting duplicate and out-of-sequence errors while copying to flat file. There is no question about it. As Peter and Bill are suspecting there might be an issue while adding/updating records into this file. I found that only 2 CICS programs are adding/updating the file. Most importantly this file being used by 2 different CICS regions and that's the reason share options (3,3).

Anyway I am going to look into these 2 programs what it is doing. I will not give up until this is solved of course you guys will be there to help me.

Lastly can anybody please explain me this:- VSAM should be handling out-of-sequence and duplicate conditions when record is getting added even if SHR(3,3). Why its not happening?. or anything wrong with cluster definition or something else?

Thanks,
Chetan
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Tue Jul 17, 2012 8:10 pm
Reply with quote

If you have two processes updating the same VSAM file at the same time you will make a mess of the file unless you use a seperaate method to control which process updates when.

If one has just read an index, the second reads the same index, the second inserts a record, the first inserts a record, then your index has been compromised, and possibly your data as well.

Without looking in detail at what particular sequence of events caused a particular problem, you won't be able to predict the outcome beyond how Peter has already described it. It's that one begining with the letter F.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Tue Jul 17, 2012 8:15 pm
Reply with quote

this info may help you fix the dataset Broken Dataset

as far as the problem is concerned,
there is a little blurb in VSAM Dym... (page 153) that states:
Quote:
 Sharing a data set across regions (cross-region) or across systems
(cross-system) without using proper enqueuing procedures to protect data set
integrity. For shareoptions shr(3 3) shr(4 3) shr(3 4) see related information in
OY36328.
 Sharing a data set across systems — even using the appropriated share
option — but without propagating ENQ name SYSVSAM around the GRS
ring. This is the *most* common user error. It results in duplicate index
pointers in the high level index records. See message IDC11705I
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA

PostPosted: Tue Jul 17, 2012 8:25 pm
Reply with quote

Quote:
VSAM should be handling out-of-sequence and duplicate conditions when record is getting added even if SHR(3,3).
Sorry, but you are making a false statement here. From the AMS for Catalogs manual on SHROPTIONS (with emphasis added by me):
Quote:

3
Specifies that the data set can be fully shared. With this option, each user is responsible for maintaining both read and write integrity for the data that user's program accesses. User programs that ignore write integrity guidelines can cause VSAM program checks, uncorrectable data set errors, and other unpredictable results. This option requires each user to be responsible for maintenance. The RESERVE and DEQ macros are required with this option to maintain data set integrity. (For information on using RESERVE and DEQ, see z/OS MVS Programming: Authorized Assembler Services Reference ALE-DYN and z/OS MVS Programming: Authorized Assembler Services Reference LLA-SDU.)

In other words, your site has caused this mess by not following IBM standards -- you're going to have to either change the SHROPTIONS or add enqueue / dequeue / reserve logic to the programs in each region to stop the problems -- or they will continue.
Back to top
View user's profile Send private message
vasanthkumarhb

Active User


Joined: 06 Sep 2007
Posts: 275
Location: Bang,iflex

PostPosted: Tue Jul 17, 2012 10:21 pm
Reply with quote

Hi.

Quote:
Vasanth,

Yes.. there are more steps in my job. This is what it doing:-
Step1: Copy VSAM fiel to flat file using REPRO.
Step2: Process the flat file and create o/p file out of it.
Step 3: Delete and Define the VSAM.
Step 4: Load data from o/p of step 2 to VSAM file using REPRO (with REPLACE option).


By the by first tell me on which step you are getting sequence error.

Points below can encounter:-

1. Check whether you correctly sorting on key field also on mentioned length of the key..
2. If this error arriving on every week or 2 months. Try to have powerful key after discussing with your peers. Also verify your input shouldn't vary on these days.
3. Since two CICS region involved adding/updating on VSAM file. verify any user adding messy data.
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA

PostPosted: Tue Jul 17, 2012 10:38 pm
Reply with quote

Vasanth, the VSAM shareoptions are (3 3) -- unless all programs accessing the VSAM file in both CICS regions use enqueue / dequeue mechanisms, there is no need for "messy data" to have problems. Merely having both regions reading and writng to the VSAM file without using a queueing mechanism can cause the problems described.

The sort doesn't matter -- and could even be right at all times.
The user input doesn't matter -- and could even be right at all times.
The data quality doesn't matter -- and all data may be perfectly good at all times.
The only thing that matters is whether or not all programs accessing the VSAM file in both CICS regions use the proper way to ensure read and write integrity. If they don't, errors WILL occur.
Back to top
View user's profile Send private message
chetanambi

New User


Joined: 21 Jan 2012
Posts: 58
Location: India

PostPosted: Tue Jul 17, 2012 11:56 pm
Reply with quote

After reading Bill's description and info provided by dbzTHEdinosauer and Robert, now I am clear about what is causing the issue. This is indeed an issue with improper use of share-options or not following IBM standards on using ENQ/DEQ/RESERVE logic in the programs.

As of now I am not aware these commands on how to use in my CICS application. So I am going thru "VSAM demystified" to get clear picture. Please let me know if you know any useful links for these.

I will reply as I get any progress. I am very happy that even though issue is not resolved I have already learnt many things on VSAM.

Thanks,
Chetan
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA

PostPosted: Wed Jul 18, 2012 1:22 am
Reply with quote

In addition to VSAM Demystified, look at the CICS Application Programming Reference manual for the EXEC CICS ENQ and EXEC CICS DEQ commands.
Back to top
View user's profile Send private message
Bill O'Boyle

CICS Moderator


Joined: 14 Jan 2008
Posts: 2501
Location: Atlanta, Georgia, USA

PostPosted: Wed Jul 18, 2012 2:08 am
Reply with quote

Yup, after you get this fixed, it will happen again.

Alternatively, building/defining an FOR (File Owning Region) and defining this file in an FCT entry (along with others which are accessed similarly) would be beneficial in the long run.

If you go with the FOR, each current FCT entry in each given region (AOR), must specify the KEYLENGTH value, providing the program's file API doesn't, along with the SYSID of the FOR (REMOTESYSTEM). Without a KEYLENGTH in the API as well as being omitted from the FCT entry, an INVREQ will be raised. With that, it's much easier to specify the KEYLENGTH value along with the mandatory REMOTESYSTEM/SYSID value in the FCT entry.

FWIW, when restarting regions, the FOR is always the first one....
Back to top
View user's profile Send private message
chetanambi

New User


Joined: 21 Jan 2012
Posts: 58
Location: India

PostPosted: Wed Jul 18, 2012 12:00 pm
Reply with quote

Bill,
Even after changing the share-options or using those CICS commands to ensure read and write integrity, problem will occur again??. Is that what you want to say?

I am not very clear about other part of your reply. The VSAM file is already defined in both the CICS regions with KEYLENGTH and I have verfied this thru CEMT command. This is what I see there:-
Code:

File(xxxxxxx)                                         
Accessmethod(Vsam)                                   
Openstatus( Open )                                   
Enablestatus( Enabled )                               
Readstatus( Read )                                   
Updatestatus( Update )                               
Addstatus( Addable )                                 
Browsestatus( Browse )                               
Deletestatus( Delete )                               
Exclstatus(              )                           
Disposition( Share )                                 
Rlsaccess( Notrls )                                   
Emptystatus( Noemptyreq )                             
Dsname( XXXXX.XX.XXXX.XXXXXXX                      )
Table( Nottable )                                     
Loadtype( Noload )                                   
Cfdtpool(          )                                 
Tablename(          )                               
Updatemodel( Locking )         
Maxnumrecs( 00000000 )         
Keylength( 014 )               
Recordsize( 00106 )           
Rbatype(Notextended)           
Remotesystem()                 


Thanks,
Chetan
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed Jul 18, 2012 12:14 pm
Reply with quote

No, I'm not saying that.

Your shareoptions allows multiple concurrent opens-for-update.

If you change your shareoptions to allow only one open-for-update at a time, you won't get the problem (but won't have simultaneous access in two CICS regions).

If you do the enqueing process as outlined by people above then you can have the file open in multiple programs but you will have a method whereby only one update of any type (add/change/delete) can be done at any one time (presuming coded correctly). As only one update can be done at a time, there is no possibility of "losing" an chunk of the index through two or more updates being done at the same time.

Don't get worried about "performance". It is pretty zippy and it is the way everyone does it and it has to be done.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Wed Jul 18, 2012 1:17 pm
Reply with quote

Bill O'Boyle wrote:
Yup, after you get this fixed, it will happen again.

Alternatively, building/defining an FOR (File Owning Region) and defining this file in an FCT entry (along with others which are accessed similarly) would be beneficial in the long run.


hmmmm, I wonder why Bill brought this up?

probably because when you have an enq/deq from every request
within an Appllication Owning Region,
having two regions with implicit FOR (as your site is set-up)
is the same as having no enq/deq.

you can only effectively enq/deq a resource one place,
in this case an FOR.

hmmmm, wonder why IBM came up with the idea of AOR and FOR?

Hmmm???

can both regions have cause to access the same records?
if not, maybe divide the VSAM into two VSAMs based on the logic driving each region.
Then you don't have to have you storage management people learn about AOR and FOR.
Back to top
View user's profile Send private message
Anuj Dhawan

Superior Member


Joined: 22 Apr 2006
Posts: 6250
Location: Mumbai, India

PostPosted: Wed Jul 18, 2012 8:06 pm
Reply with quote

This thread is a keeper - but, why not redefine VSAM with SHR(2 3)? If you want to do batch update processing on a VSAM file that is used by CICS, use CEMT to close it within CICS, so allowing a batch job to be the single permitted writer, no?
Back to top
View user's profile Send private message
chetanambi

New User


Joined: 21 Jan 2012
Posts: 58
Location: India

PostPosted: Wed Jul 18, 2012 8:24 pm
Reply with quote

I have gone thru some of the links provided. For the devices shared between multiple z/OS systems, enqueuing is accomplished though RESERVE and DEQ. and if one system ENQ and DEQ can be used. I know about ENQ and DEQ commands but didn't find any info on RESERVE from the net. Please provide me if you have any information on RESERVE. or if I use ENQ or DEQ commands is that ok?

Thanks,
Chetan
Back to top
View user's profile Send private message
PeterHolland

Global Moderator


Joined: 27 Oct 2009
Posts: 2481
Location: Netherlands, Amstelveen

PostPosted: Wed Jul 18, 2012 8:26 pm
Reply with quote

Reserve :


MVS Programming: Authorized Assembler Services Reference, Volume 3 (LLACOPY-SDUMPX)
Back to top
View user's profile Send private message
Bill O'Boyle

CICS Moderator


Joined: 14 Jan 2008
Posts: 2501
Location: Atlanta, Georgia, USA

PostPosted: Wed Jul 18, 2012 8:33 pm
Reply with quote

If you have more than one region attempting to UPDATE and you're using ENQ/DEQ, they will only be known in the region issuing the UPDATE. In other words, REGIONA won't know about REGIONB's ENQ/DEQ's because they're not GLOBAL/SHAREABLE within an LPAR or two or more REGION's.

This is why an FOR (in the same LPAR) could centralise access and give you better file integrity.
Back to top
View user's profile Send private message
chetanambi

New User


Joined: 21 Jan 2012
Posts: 58
Location: India

PostPosted: Wed Jul 18, 2012 9:08 pm
Reply with quote

Quote:

This thread is a keeper - but, why not redefine VSAM with SHR(2 3)? If you want to do batch update processing on a VSAM file that is used by CICS, use CEMT to close it within CICS, so allowing a batch job to be the single permitted writer, no?

No we can't redefine file with SHR(2,3) because dataset should be fully shared among number of users (but the programs which are adding/updating the records in the file are not maintaing the read and write integrity and is causing the issue). Howewer we did try with SHR(2,3) sometime back and it worked for somedays but issue started occuring again after that.

Thanks,
Chetan
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Wed Jul 18, 2012 10:23 pm
Reply with quote

Hello,

Probably not what you are looking for, but maybe it is time to convert this to a database. . .

And we appear to have gone far away from the "out of sequence" icon_smile.gif

d
Back to top
View user's profile Send private message
chetanambi

New User


Joined: 21 Jan 2012
Posts: 58
Location: India

PostPosted: Thu Jul 19, 2012 12:56 pm
Reply with quote

Quote:
If you have more than one region attempting to UPDATE and you're using ENQ/DEQ, they will only be known in the region issuing the UPDATE. In other words, REGIONA won't know about REGIONB's ENQ/DEQ's because they're not GLOBAL/SHAREABLE within an LPAR or two or more REGION's

Bill, Yes. more than one regions (in our case 2 regions) may try to update the data. Since more than one regions are involved here how to make it GLOBAL/SHAREABLE?. I found some information in VSAM demystified about GSR (Global Shared Resource) but no explanation how to use. Is that what you mean by GLOBAL/SHAREABLE?. Could you please explain me how can i use it for better file integrity?

So I am gonna try by changing share-options(2,3) once more time AND/OR with GLOBAL/SHAREABLE after I get information on it.

I have noticed that CEMT command for file is showing RLSACCESS as below. How about using the file in RLS (Record Level Sharing) mode across LPAR. Will that solve the problem?
Code:
Rlsaccess( Notrls )
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM Goto page 1, 2, 3  Next

 


Similar Topics
Topic Forum Replies
No new posts How to split large record length file... DFSORT/ICETOOL 7
No new posts Error to read log with rexx CLIST & REXX 11
No new posts Extracting Variable decimal numbers f... DFSORT/ICETOOL 17
No new posts Error when install DB2 DB2 2
No new posts SFTP Issue - destination file record ... All Other Mainframe Topics 2
Search our Forums:

Back to Top