RECORD OUT OF SEQUENCE error on VSAM KSDS file..

expat · Posted: Tue Jul 17, 2012 4:56 pm

Peter, good point.

I have just noticed SHR(3 3) on the cluster definition

chetanambi · New User Joined: 21 Jan 2012 Posts: 58 Location: India

I did try with EXEC PGM=IDCAMS but the result is same

..

Ya.. in this case both VERIFY and REPRO are catching if there is something wrong with the file. I am really getting duplicate and out-of-sequence errors while copying to flat file. There is no question about it. As Peter and Bill are suspecting there might be an issue while adding/updating records into this file. I found that only 2 CICS programs are adding/updating the file. Most importantly this file being used by 2 different CICS regions and that's the reason share options (3,3).

Anyway I am going to look into these 2 programs what it is doing. I will not give up until this is solved of course you guys will be there to help me.

Lastly can anybody please explain me this:- VSAM should be handling out-of-sequence and duplicate conditions when record is getting added even if SHR(3,3). Why its not happening?. or anything wrong with cluster definition or something else?

Thanks,
Chetan

Bill Woodger · Posted: Tue Jul 17, 2012 8:10 pm

If you have two processes updating the same VSAM file at the same time you will make a mess of the file unless you use a seperaate method to control which process updates when.

If one has just read an index, the second reads the same index, the second inserts a record, the first inserts a record, then your index has been compromised, and possibly your data as well.

Without looking in detail at what particular sequence of events caused a particular problem, you won't be able to predict the outcome beyond how Peter has already described it. It's that one begining with the letter F.

dbzTHEdinosauer · Posted: Tue Jul 17, 2012 8:15 pm

this info may help you fix the dataset Broken Dataset

as far as the problem is concerned,
there is a little blurb in VSAM Dym... (page 153) that states:

Robert Sample · Posted: Tue Jul 17, 2012 8:25 pm

vasanthkumarhb · Posted: Tue Jul 17, 2012 10:21 pm

Hi.

Robert Sample · Posted: Tue Jul 17, 2012 10:38 pm

Vasanth, the VSAM shareoptions are (3 3) -- unless all programs accessing the VSAM file in both CICS regions use enqueue / dequeue mechanisms, there is no need for "messy data" to have problems. Merely having both regions reading and writng to the VSAM file without using a queueing mechanism can cause the problems described.

The sort doesn't matter -- and could even be right at all times.
The user input doesn't matter -- and could even be right at all times.
The data quality doesn't matter -- and all data may be perfectly good at all times.
The only thing that matters is whether or not all programs accessing the VSAM file in both CICS regions use the proper way to ensure read and write integrity. If they don't, errors WILL occur.

chetanambi · New User Joined: 21 Jan 2012 Posts: 58 Location: India

After reading Bill's description and info provided by dbzTHEdinosauer and Robert, now I am clear about what is causing the issue. This is indeed an issue with improper use of share-options or not following IBM standards on using ENQ/DEQ/RESERVE logic in the programs.

As of now I am not aware these commands on how to use in my CICS application. So I am going thru "VSAM demystified" to get clear picture. Please let me know if you know any useful links for these.

I will reply as I get any progress. I am very happy that even though issue is not resolved I have already learnt many things on VSAM.

Thanks,
Chetan

Robert Sample · Posted: Wed Jul 18, 2012 1:22 am

In addition to VSAM Demystified, look at the CICS Application Programming Reference manual for the EXEC CICS ENQ and EXEC CICS DEQ commands.

Bill O'Boyle · Posted: Wed Jul 18, 2012 2:08 am

Yup, after you get this fixed, it will happen again.

Alternatively, building/defining an FOR (File Owning Region) and defining this file in an FCT entry (along with others which are accessed similarly) would be beneficial in the long run.

If you go with the FOR, each current FCT entry in each given region (AOR), must specify the KEYLENGTH value, providing the program's file API doesn't, along with the SYSID of the FOR (REMOTESYSTEM). Without a KEYLENGTH in the API as well as being omitted from the FCT entry, an INVREQ will be raised. With that, it's much easier to specify the KEYLENGTH value along with the mandatory REMOTESYSTEM/SYSID value in the FCT entry.

FWIW, when restarting regions, the FOR is always the first one....

chetanambi · New User Joined: 21 Jan 2012 Posts: 58 Location: India

Bill,
Even after changing the share-options or using those CICS commands to ensure read and write integrity, problem will occur again??. Is that what you want to say?

I am not very clear about other part of your reply. The VSAM file is already defined in both the CICS regions with KEYLENGTH and I have verfied this thru CEMT command. This is what I see there:-

Bill Woodger · Posted: Wed Jul 18, 2012 12:14 pm

No, I'm not saying that.

Your shareoptions allows multiple concurrent opens-for-update.

If you change your shareoptions to allow only one open-for-update at a time, you won't get the problem (but won't have simultaneous access in two CICS regions).

If you do the enqueing process as outlined by people above then you can have the file open in multiple programs but you will have a method whereby only one update of any type (add/change/delete) can be done at any one time (presuming coded correctly). As only one update can be done at a time, there is no possibility of "losing" an chunk of the index through two or more updates being done at the same time.

Don't get worried about "performance". It is pretty zippy and it is the way everyone does it and it has to be done.

dbzTHEdinosauer · Posted: Wed Jul 18, 2012 1:17 pm

Anuj Dhawan · Posted: Wed Jul 18, 2012 8:06 pm

This thread is a keeper - but, why not redefine VSAM with SHR(2 3)? If you want to do batch update processing on a VSAM file that is used by CICS, use CEMT to close it within CICS, so allowing a batch job to be the single permitted writer, no?

chetanambi · New User Joined: 21 Jan 2012 Posts: 58 Location: India

I have gone thru some of the links provided. For the devices shared between multiple z/OS systems, enqueuing is accomplished though RESERVE and DEQ. and if one system ENQ and DEQ can be used. I know about ENQ and DEQ commands but didn't find any info on RESERVE from the net. Please provide me if you have any information on RESERVE. or if I use ENQ or DEQ commands is that ok?

Thanks,
Chetan

PeterHolland · Posted: Wed Jul 18, 2012 8:26 pm

Reserve :

MVS Programming: Authorized Assembler Services Reference, Volume 3 (LLACOPY-SDUMPX)

Bill O'Boyle · Posted: Wed Jul 18, 2012 8:33 pm

If you have more than one region attempting to UPDATE and you're using ENQ/DEQ, they will only be known in the region issuing the UPDATE. In other words, REGIONA won't know about REGIONB's ENQ/DEQ's because they're not GLOBAL/SHAREABLE within an LPAR or two or more REGION's.

This is why an FOR (in the same LPAR) could centralise access and give you better file integrity.

chetanambi · New User Joined: 21 Jan 2012 Posts: 58 Location: India

dick scherrer · Posted: Wed Jul 18, 2012 10:23 pm

Hello,

Probably not what you are looking for, but maybe it is time to convert this to a database. . .

And we appear to have gone far away from the "out of sequence"

d

chetanambi · New User Joined: 21 Jan 2012 Posts: 58 Location: India

Anuj Dhawan · Posted: Thu Jul 19, 2012 1:30 pm

dbzTHEdinosauer · Posted: Thu Jul 19, 2012 2:07 pm

be careful about saying enq/deq and batch.

one of the reasons most batch activity is performed on a VSAM file after the file has been closed and disabled in cics
is due to the fact that CICS does not always immediately update the physical file,
but instead maintains the updated CI in the buffer.
and until the file is CLOSED in CICS, the buffers are not flushed.
the file also needs to be disabled in CICS is to prohibit re-opening the file in cics
by any transaction that may attempt to read the file,
is the file is defined by the FCT to allow a read to open an closed/enabled file.

also, since enq/deq are part of the CICS API's,
the only way a Batch program could enq/deq a CICS resource
is to invoke a CICS module in an active CICS Region
(via CICS LINK - exci or any other batch to cics interface)
to perform the enq/deq
or if the interface supports the enq/deq API directly.

also, keep in mind, that the enq/deq is for a resource name,
and unless all enq/deq (for a vsam file in this case) use the same resource name (or id)
then effectively there is no enq/deq of a resource.

good post Anuj, allows for misconceptions to be addressed.

expat · Posted: Thu Jul 19, 2012 2:13 pm

You could investigate the use of DFSMStvs at your site, although that would be a pay for addition from IBM. It allows simultaneous multiple access to the same VSAM file through both batch and online.

Another possibility may be RLS via SMS but you would need to talk to your storage management / sysprogs about it.

dbzTHEdinosauer · Posted: Thu Jul 19, 2012 3:07 pm

db2 databases are the only data repositories that allow for concurrent batch and CICS update.

even though DFSMStvs with RLS allows and protects the integrity of VSAM datasets
during concurrent multi-CICS-Region updates,
this does not include batch.
the literature states:
a batch job updating a vsam dataset
must be disabled/closed and then re-opened in CICS in a non-RLS mode.

there maybe 3rd party tools (read more bucks) that allow for concurrent batch and CICS updates,
but, personally, i doubt it. CICS controls vsam datasets in cics buffers,
even dfsmstvs functions under CICS.
and since these buffer can/will contain modified data that will not be positively written to the dataset until disable/close,
these buffers are not available to batch.

the only way around this problem,
would to have the batch update interfacing with something like exci
which means that the updating would be performed by CICS
and not a batch allocation of the file.

as with all cics vsam enhancements to allow for 24/7,
there are caveats,
the biggest hurdle being you can't update an open cics vsam file in RLS mode from a batch allocation of the file ....

the solution provided by Bill o'boyle concerning AOR/FOR allows
for the easiest method of implementing data integrity in CICS.
But, even that does not address the problem of concurrent batch/cics update.

by the time you add RLS (required for dfsmstvs) and dfsmstvs to your system
(normally, more iron required)
you are starting to enter the point where your scale of cost
approaches DB2,
which allows for concurrent batch/cics update
mainly because CICS does not control db2, as it does control VSAM.

dbzTHEdinosauer · Posted: Thu Jul 19, 2012 3:18 pm

I got out of the vsam business long ago (and migrated to db2)
because there is no growth future.
the more you add to a vsam system, especially the 24/7 requirements,
you are contantly fighting the same battles,
and the TS is plagued with a CICS/datamanagment support group
that does not even know the basics.
otherwise the TS would not have created this thread.

another reason for the direct vsam image to db2 row conversions.
they were basically trying to get around the problem inherent in vsam
with 24/7 online and batch updating
on the cheap (read direct conversion of vsam record to db2 row).