This task fails after around an hour and a half, with the following error -
IDC3302I ACTION ERROR ON TEST.N3F00.CREMIDAA.G0001V00
IDC3309I ** RECORD X'F0F0F0C7E4' NOT WRITTEN. LENGTH INVALID
IDC3302I ACTION ERROR ON TEST.N3F00.CREMIDAA.G0001V00
IDC3309I ** RECORD X'F0F0F0C7D2' NOT WRITTEN. LENGTH INVALID
IDC3302I ACTION ERROR ON TEST.N3F00.CREMIDAA.G0001V00
IDC3309I ** RECORD X'F0F0F0C7D2' NOT WRITTEN. LENGTH INVALID
IDC3302I ACTION ERROR ON TEST.N3F00.CREMIDAA.G0001V00
IDC3309I ** RECORD X'F0F0F0C7D2' NOT WRITTEN. LENGTH INVALID
IDC31467I MAXIMUM ERROR LIMIT REACHED.
IDC0005I NUMBER OF RECORDS PROCESSED WAS 1595555
IDC3003I FUNCTION TERMINATED. CONDITION CODE IS 12
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 12
Useful Information -
1595555 records are the number of records in the First Input Dataset.
I am almost positive that the Record-Length phrase in the error message doesnt have anything to do with the Record Length of the 2nd or the 3rd Input Tape Dataset. If that was to be true, the Concatenation should have thrown an error (I could be wrong!). The following JCL I used to create the Input Tape Datasets should make it clear that the Datasets are of the correct LRECL -
Besides, even if we get this to work somehow, the amount of time it takes now (1.5 hours is UNACCEPTABLE). The same process completes in our Production environment in 18 minutes, except that in Production we have NO DD CONCATENATION. There is just one Input Tape Dataset.
The Concatentation together has around 7.4 Million records.
Gurus please rescue!!! This is REAL URGENT!!!!!
Joined: 23 Nov 2006 Posts: 19243 Location: Inside the Matrix
Hello,
Check the dcb info for all 3 files.
Even if the lrecl is the same, blksize could be what is causing the error.
Make sure the first file in the concatenation is the file with the largest blksize. If you need the files in the order you've specified, then create an empty dasd dataset with a blksize equal to or larger than the largest of the 3 files you need to input.
Joined: 23 Nov 2006 Posts: 19243 Location: Inside the Matrix
Hello,
Also, was there a reason to combine the files with IDCAMS rather than IEBGENER or SORT? SORT will usually copy files faster than either IDCAMS or SORT.
Thanks a lot for the quick reply .. that was real fast!!!
I am in the process of reallocating & repopulating these files with all the 7.4 Million records. Just this time, I am going to make sure I have the Block sizes of all them set to 1000.
Since this is going to take some time, I will get back in some time.
Thanks again! You will hear from me regardless of whether this solves the problem.
Frank per your suggestion I ran the following JCL -
//TIBNRCPY JOB (Acct Programer)
// 'COPY IBNR IP',
// CLASS=R,
//*
//CPYIBNR EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=($,,STAN)
//DFSMSG DD SYSOUT=($,,STAN)
//INDD DD DSN=TEST.N3F00.CREMIDAA.INF.M0000,DISP=OLD,
// DCB=(RECFM=FB,LRECL=1000,BLKSIZE=32000)
// DD DSN=TEST.N3F00.CREMIDAA.INF.M0001,DISP=SHR
// DD DSN=TEST.N3F00.CREMIDAA.INF.M0002,DISP=SHR
//OUTDD DD DSN=TEST.N3F00.CREMIDAA(+1),
// DISP=(NEW,CATLG,DELETE),VOL=(,,,20),
// UNIT=T3480,LABEL=RETPD=010,
// DCB=(SYS3.DSCB,RECFM=FB,LRECL=1000,BLKSIZE=0)
//TOOLIN DD *
COPY FROM(INDD) TO(OUTDD)
/*
I know I was sounding as if I am going to delete and reallocate and repopulate the 3 files with all the data, but turns out that it is going to take too much time. Meanwhile, I thought of giving this a try.
This Job continues to take a lot of time.
What could be wrong? It has been 25 minutes and there are no signs of this Job finishing.
Just so you know, I tried running the Job without the DCB on the First Input Dataset used for concatenation. It failed with a User ABEND code U016 saying that INDD had different RECFM than OUTDD viz., INDD had RECFM "U". That comes as REAL SURPRISE knowing the JCL I used to create the Tape Dataset.
Joined: 23 Nov 2006 Posts: 19243 Location: Inside the Matrix
Hi Leon,
If you specified the 32k blksize for your INPUT data, it will not improve the performance as the actual file(s) are blocked lower. The "undefined" format does look strange, but see below.
Do you still have the sysout form the job(s) that created the 3 files you want to use as input? If you can post them, that may help.
If these are all "tape" files, i'd hope the tape management system has information about the files. Please look at (or ask the folks who do tape support to help) the information about the 3 input files. This is just to make sure they really do have the expected dcb info.
Did this file
Code:
//OUTDD DD DSN=TEST.N3F00.CREMIDAA(+1),
already exist as a gdg in your system? If it did not, you will first have to define the gdg to the system. This may well be what caused the "undefined". When you specified the "SYS3.DSCB" and the definition was not there, that may well have yielded recfm=u.
For tonight, i'd suggest re-running the job with a totally different output dataset name - and not one that is the high level name of the other datasets and not a new gdg. I believe the jcl used was Frank's example - not meant to literally be the jcl to submit. This output file WILL bebefit from the larger blksize.
Here's the Mainframe SYSOUT for the 2 Tape Input files -
For File INF.M000
DATA SET UTILITY - GENERATE
WER164B 10,844K BYTES OF VIRTUAL STORAGE AVAILABLE, MAX REQUESTED,
WER164B 64K BYTES RESERVE REQUESTED, 242,104 BYTES USED
WER146B 20K BYTES OF EMERGENCY SPACE ALLOCATED
WER108I SYSUT1 : RECFM=FB ; LRECL= 1000; BLKSIZE= 1000
WER110I SYSUT2 : RECFM=FB ; LRECL= 1000; BLKSIZE= 32000
WER211B SYNCSMF CALLED BY SYNCSORT; RC=0000
WER449I SYNCSORT GLOBAL DSM SUBSYSTEM ACTIVE
WER416B BSAM WAS USED FOR SYSUT1
WER416B BSAM WAS USED FOR SYSUT2
WER054I RCD IN 0, OUT 0
WER169I RELEASE 1.1D BATCH 0414 TPF LEVEL 3
WER052I END SYNCSORT - TNF4IBCM,STEP01,N3FCRIM2,DIAG=8C00,CA82,A08E,6C6C,C472,
SYNCSORT COPY FEATURE CALLED - RC = 0
For File INF.M0001
DATA SET UTILITY - GENERATE
WER164B 10,844K BYTES OF VIRTUAL STORAGE AVAILABLE, MAX REQUESTED,
WER164B 64K BYTES RESERVE REQUESTED, 242,104 BYTES USED
WER146B 20K BYTES OF EMERGENCY SPACE ALLOCATED
WER108I SYSUT1 : RECFM=FB ; LRECL= 1000; BLKSIZE= 1000
WER110I SYSUT2 : RECFM=FB ; LRECL= 1000; BLKSIZE= 32000
WER211B SYNCSMF CALLED BY SYNCSORT; RC=0000
WER449I SYNCSORT GLOBAL DSM SUBSYSTEM ACTIVE
WER416B BSAM WAS USED FOR SYSUT1
WER416B BSAM WAS USED FOR SYSUT2
WER054I RCD IN 0, OUT 0
WER169I RELEASE 1.1D BATCH 0414 TPF LEVEL 3
WER052I END SYNCSORT - TNF4IBCM,STEP02,N3FCRIM2,DIAG=8600,C004,AA82,E866,CE7E,
SYNCSORT COPY FEATURE CALLED - RC = 0
For INF.M0002
DATA SET UTILITY - GENERATE
WER164B 10,844K BYTES OF VIRTUAL STORAGE AVAILABLE, MAX REQUESTED,
WER164B 64K BYTES RESERVE REQUESTED, 242,104 BYTES USED
WER146B 20K BYTES OF EMERGENCY SPACE ALLOCATED
WER108I SYSUT1 : RECFM=FB ; LRECL= 1000; BLKSIZE= 1000
WER110I SYSUT2 : RECFM=FB ; LRECL= 1000; BLKSIZE= 32000
WER211B SYNCSMF CALLED BY SYNCSORT; RC=0000
WER449I SYNCSORT GLOBAL DSM SUBSYSTEM ACTIVE
WER416B BSAM WAS USED FOR SYSUT1
WER416B BSAM WAS USED FOR SYSUT2
WER054I RCD IN 0, OUT 0
WER169I RELEASE 1.1D BATCH 0414 TPF LEVEL 3
WER052I END SYNCSORT - TNF4IBCM,STEP03,N3FCRIM2,DIAG=A400,628E,888A,60C4,EC76,
SYNCSORT COPY FEATURE CALLED - RC = 0
Joined: 23 Nov 2006 Posts: 19243 Location: Inside the Matrix
Hi Leon,
The copy jobs look good. The new output files should have the 32k blksize (per the output you posted).
However. . . . The output also shows that you are running SYNCSORT - not the IBM sort (DFSORT). The ICETOOL is part of DFSORT, not SYNCSORT.
It also appears that your site has implemented the "trick" that when a straight copy by "IEBGENER" is detected, the sort is executed instead for better performance.
If you concatenate the 3 "new" output/input files and run another step, you should get the single file you are looking for. . .
I finally have gotten into deleting - reallocating and repopulating the Input Tape Datasets with data - 7.4 Million Records.
I noted one difference in my Utility that populates these 3 Datasets with data against it's Production Counterpart. I have changed my Test Utility so that that's taken care of. I am waiting to see how this helps.
The Block Size of all three files is now 32000 as seen in the Dataset allocation Job.
I will let you know how I make out. I am just hoping it works.
This finally got resolved. I know you think I must be crazy when I say this, but it looks like it was in the difference I had in the Utitity that fills data in the Datasets that caused the problem. I hadn't put the Tape Dataset name in quotes the last time when I ran the uploads. This was different from Production where in the Utility had quotes in it for the Dataset name.
Wonder how this could have caused the dataset already allocated to have a RECFM of FB to become U?
Spare me my ignorance!
Dick, Frank, I really appreciate all your help in this regard!!
Joined: 23 Nov 2006 Posts: 19243 Location: Inside the Matrix
Hi Leon,
One thing that happens with some processes that receive a dsn "without quotes" is that "your id" is placed before the rest of the value you specify.
Go into tso and allocate a dataset via 3.2 - specify the dsn as MYTEST then hit enter. Notice that there is a node added before MYTEST on the Allocate New Data Set screen. PF3 out to prevent actual allocation. Now do this again but specify 'MYTEST' and notice that only MYTEST is in the dsn. This is probably what happened in your case.
