I have below Sort step running in production and I have seen some records are dropped during the sort process from file1 which they shouldn't have based on their actual file1 values. So to verify , I ran all the same steps with production files in Development region and I got the correct results which means no records got dropped but production one has missing records. What could have gone wrong , any idea?
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
Why use a SORT operator with a USING which contains SORT FIELDS=COPY? Just use the COPY operator.
You are SORTing both files for the JOIN, do you need to? Why don't you have JNFnCNTL files for the JOIN put INCLUDE/OMIT COND= in there, and also, since you only need the keys for the F2, just BUILD the key,
The SORT of TEMP3 you could do in the JOINKEYS main task. I have no clue as to why you do the final SORT (COPY).
As to if/why records may be missing, we have nothing to go on, as there is no information of what is missing, or even the sysout from the step missing and with.
I agree we can rewrite this better with your observation but do you see any strange could happen here and this in production since last 4 years? Conditions in mem2 and mem3 are satisfied for those 10 records which shouldn't have skipped from file to out1 in production. Input records count is around 12k. but the same things works fine in development region.
Thanks Bill.
It was a scheduling issue where the jobs are not scheduled properly. The file1 referenced above is created in job1 and the above step executes in job2Now in this case the Job1 ran 1 minute after job2 which then made job2 to refer file1's yesterday's generation where
Code:
3500,1,CH,NE,C' '
wasn't true hence the results and during the testing in development region I was using current (Today's generation of Job1) as this was how scheduled in production and now this has been corrected.
