IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Drop all the records before a criteria


IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Tue May 14, 2013 2:13 am
Reply with quote

Hi,

I have a file which has below specification:
Code:
Organization  . . . : PS   
Record format . . . : FB   
Record length . . . : 250 
Block size  . . . . : 27750
1st extent cylinders: 1   
Secondary cylinders : 300 
Data set name type  : 


Currently the file has multiple header records which are repeating. I am manually dropping records based on below criteria.
Step 1:
Code:
 F '01' 1 Last

Step2: Dropping all the records from top to the record found from step1.

Is it possible to do this by SORT /ICETOOL SUBSET or any other option.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Tue May 14, 2013 3:06 am
Reply with quote

Are the duplicate headers contiguous?

Is all the data on the headers the same? Or is just the last one correct?
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Tue May 14, 2013 3:14 am
Reply with quote

Yes The header, detail and the trailer are repeating.Please see below:
Code:
01Header
04Detail1
04Detail2
04Detail3
98Trailer1
99Trailer
01Header
04Detail1
04Detail2
04Detail3
98Trailer1
99Trailer
01Header
04Detail1
04Detail2
04Detail3
98Trailer1
99Trailer
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose

PostPosted: Tue May 14, 2013 3:30 am
Reply with quote

abhijit.nayak01 wrote:
Yes The header, detail and the trailer are repeating.Please see below:


And the desired output is the Last 5 records? like this

Code:

01HEADER      - DROP                   
04DETAIL1     - DROP                   
04DETAIL2     - DROP                   
04DETAIL3     - DROP                   
98TRAILER1    - DROP                   
99TRAILER     - DROP                   
01HEADER      - DROP                   
04DETAIL1     - DROP                   
04DETAIL2     - DROP                   
04DETAIL3     - DROP                   
98TRAILER1    - DROP                   
99TRAILER     - DROP                   
01HEADER      - DROP                   
04DETAIL1     - WRITE TO OUTPUT ??     
04DETAIL2     - WRITE TO OUTPUT ??     
04DETAIL3     - WRITE TO OUTPUT ??     
98TRAILER1    - WRITE TO OUTPUT ??     
99TRAILER     - WRITE TO OUTPUT ??     
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Tue May 14, 2013 3:38 am
Reply with quote

The desire output is as below:
Code:
01HEADER      - DROP                   
04DETAIL1     - DROP                   
04DETAIL2     - DROP                   
04DETAIL3     - DROP                   
98TRAILER1    - DROP                   
99TRAILER     - DROP                   
01HEADER      - DROP                   
04DETAIL1     - DROP                   
04DETAIL2     - DROP                   
04DETAIL3     - DROP                   
98TRAILER1    - DROP                   
99TRAILER     - DROP                   
01HEADER      - WRITE TO OUTPUT                   
04DETAIL1     - WRITE TO OUTPUT     
04DETAIL2     - WRITE TO OUTPUT       
04DETAIL3     - WRITE TO OUTPUT       
98TRAILER1    - WRITE TO OUTPUT     
99TRAILER     - WRITE TO OUTPUT     
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose

PostPosted: Tue May 14, 2013 4:00 am
Reply with quote

I am not going to provide you complete Job, but will give you hints about solving it.

1. Look up Joinkeys examples of using the same file with temp matching keys created. There should at least 3 topics using this trick
2. For JNF1 you would use WHEN=GROUP with begin for '01' and add a 8 byte ID at the end
3. For JNF2 you would only INCLUDE '01' records and just build the record with 8 byte seqnum starting with zero.
4. Since you are creating the temp keys to be matched, make sure you have SORTED and NOSEQCK on the matching keys.
5. Use JOIN UNPAIRED,F1,ONLY to get the desired results


Once you got the desired results post the solution here so that it helps in future
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Tue May 14, 2013 6:41 pm
Reply with quote

Hi Skolusu,

Thanks for your idea. I managed to get the desired output. Please check the below JCL.

Code:
//SORT0001 EXEC PGM=SORT                                   
//IN1 DD DSN=P2D.P2D.DT130513,DISP=SHR   
//IN2 DD DSN=P2D.P2D.DT130513,DISP=SHR   
//SORTOUT DD DSN=P2D.P2D.OUT,DISP=OLD     
//SYSOUT   DD SYSOUT=*                                     
//JNF1CNTL DD *                                           
 INREC IFTHEN=(WHEN=GROUP,BEGIN=(01,2,CH,EQ,C'01'),       
              PUSH=(251:ID=8))                             
/*                                                         
//JNF2CNTL DD *                                           
 INREC IFTHEN=(WHEN=(01,2,CH,EQ,C'01'),                   
              BUILD=(251:SEQNUM,8,ZD,START=00000000))     
/*                                                         
//SYSIN DD *                                               
 JOINKEYS F1=IN1,FIELDS=(251,8,A)                         
 JOINKEYS F2=IN2,FIELDS=(251,8,A)                         
 JOIN UNPAIRED,F1,ONLY                                     
 OPTION COPY                                               
/*   
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Tue May 14, 2013 7:49 pm
Reply with quote

Well, you've got the result, but that is not always the end of it.

Have a look again at Kolusu's points 3 and 4.

In your version, you are sorting the entire file twice, with OPTION EQUALS, to get the two files into the order in which they already are.

You have records on F2 which you do not need. Although they don't do any "harm" directly, effort is still spent in processing them to be ignored.
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Wed May 15, 2013 12:15 am
Reply with quote

Hi Bill,
Please clarify more in detail. Do I need to create a temporary file and then use it for JOIN.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed May 15, 2013 3:33 am
Reply with quote

Skolusu wrote:
[...]
3. For JNF2 you would only INCLUDE '01' records and just build the record with 8 byte seqnum starting with zero.
4. Since you are creating the temp keys to be matched, make sure you have SORTED and NOSEQCK on the matching keys.
[...]Once you got the desired results post the solution here so that it helps in future


So, in JNF2CNTL you are not using INCLUDE, you are using IFTHEN=(WHEN=(logexp. This "works", but leaves a load of unwanted and unneeded records as input to the JOINKEYS.

So, use INCLUDE, then a simple INREC BUILD. Note the "just build" from Kolusu's point. That doesn't mean "just build it at the end of the record leaving 250 blanks lying around in front of it". Your records from JNF2 only need to have the sequence number on.

Which brings us to point 4.

You are creating group IDs on JNF1, so that file is already in sequence on the key for matching. You are creating a sequence number on JNF2, so that file is already in sequence on the key for matching as well.

The way you have coded, you are sorting both files (look at your sysout) on the keys to get them into the same order. With OPTION EQUALS.

If you read Kolusu's suggestion again, you would avoid the two sorts.
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Mon May 20, 2013 6:41 pm
Reply with quote

Hi Bill,
Thanks for the explanation and apologies for late response. Actually was on leave. Please see below and let me know whether the code is correct or not:
Code:
//JNF1CNTL DD *                                               
 INREC IFTHEN=(WHEN=GROUP,BEGIN=(01,2,CH,EQ,C'01'),           
              PUSH=(251:ID=8))                               
/*                                                           
//JNF2CNTL DD *                                               
 INCLUDE COND=(01,2,CH,EQ,C'01')                             
 INREC BUILD=(1:SEQNUM,8,ZD,START=00000000)                   
/*                                                           
//SYSIN DD *                                                 
 JOINKEYS F1=IN1,FIELDS=(251,8,A)                             
 JOINKEYS F2=IN2,FIELDS=(1,8,A),SORTED,NOSEQCK               
 JOIN UNPAIRED,F1,ONLY                                       
 OPTION COPY                                                 
/*                                                           
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Mon May 20, 2013 6:43 pm
Reply with quote

Forgot to mention that the code worked fine.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Mon May 20, 2013 6:50 pm
Reply with quote

You should put SORTED,NOSEQCK on both JOINKEYS files.

Since the code is working, I guess you have specified the LRECL for the output file in the JCL.

It is more flexible to leave off all DCB-related information from the JCL, and let DFSORT handle it.

You then add

Code:
  INREC BUILD=(1,250)


to the JOINKEYS main-task (put it after your OPTION COPY to be logical).

You then have all the information for the data in one place (the Sort Control Cards) rather than in two places (some in the Control Cards, some in the JCL).

This makes it much easier to maintain/change/understand, and much easier to copy and re-use for another requirement.
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Mon May 20, 2013 9:33 pm
Reply with quote

Thanks Bill and Skolusu. I am using the output file with DISP=OLD which is standard here.
Meanwhile saved my 60 hours of time as it was initially suggested to do with COBOL. icon_smile.gif
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Mon May 20, 2013 9:42 pm
Reply with quote

I'd still put the BUILD in, it will clarify in the Control Cards that the appended data is only temporary. Otherwise it'll confuse someone, someday. Maybe you in six months time :-)

If saving yourself time, it is also important not to charge the client for unnecessary processing (like leaving off the SORTED,NOSEQCK).
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose

PostPosted: Mon May 20, 2013 9:46 pm
Reply with quote

abhijit.nayak01 wrote:
Thanks Bill and Skolusu. I am using the output file with DISP=OLD which is standard here.
Meanwhile saved my 60 hours of time as it was initially suggested to do with COBOL. icon_smile.gif


I am assuming that it is NOT the same input dataset. If it is then you are inviting another problem using the same dataset for input and output. For a copy application, the SORTIN data set should not be the same as the SORTOUT data set or any OUTFIL data set because this can cause lost or incorrect data or unpredictable results.

You did NOT code the optimal job yet. Here are the control cards that you need. Look a the control cards and then compare it to the hints I gave earlier.

Code:

//SYSIN    DD *                                                   
  OPTION COPY                                                     
  JOINKEYS F1=INA,FIELDS=(251,8,A),SORTED,NOSEQCK                 
  JOINKEYS F2=INB,FIELDS=(001,8,A),SORTED,NOSEQCK                 
  JOIN UNPAIRED,F1,ONLY                                           
  INREC BUILD=(1,250)                                             
//*                                                               
//JNF1CNTL DD *                                                   
  INREC IFTHEN=(WHEN=GROUP,BEGIN=(1,2,CH,EQ,C'01'),PUSH=(251:ID=8))
//*                                                               
//JNF2CNTL DD *                                                   
  INCLUDE COND=(1,2,CH,EQ,C'01')                                   
  INREC BUILD=(SEQNUM,8,ZD,START=0)                               
//*
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Tue May 21, 2013 1:48 am
Reply with quote

Hi Bill/Skolusu,
The input and output are different as per my previous JCL.
Code:
//IN1 DD DSN=P2D.D130519,DISP=SHR                   
//IN2 DD DSN=P2D.D130519,DISP=SHR                   
//SORTOUT DD DSN=P2D.DT130519.OP,DISP=OLD           
//SYSOUT   DD SYSOUT=*                             
//JNF1CNTL DD *                                     
 INREC IFTHEN=(WHEN=GROUP,BEGIN=(01,2,CH,EQ,C'01'),
              PUSH=(251:ID=8))                     
/*                                                 
//JNF2CNTL DD *                                     
 INCLUDE COND=(01,2,CH,EQ,C'01')                   
 INREC BUILD=(1:SEQNUM,8,ZD,START=00000000)         
/*                                                 
//SYSIN DD *                                       
 JOINKEYS F1=IN1,FIELDS=(251,8,A),SORTED,NOSEQCK   
 JOINKEYS F2=IN2,FIELDS=(1,8,A),SORTED,NOSEQCK     
 JOIN UNPAIRED,F1,ONLY                             
 OPTION COPY                                       
/*                                                 




Now my only question is even if the output datset is old and it has lrecl=250 do we need
Code:
INREC BUILD=(1,250) 
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Tue May 21, 2013 4:07 am
Reply with quote

Do you "need" it in terms of the process doing what you want? No. The JCL will be providing 250 for the LRECL, so the data will be truncated to 250, and the extended ID will never have a permanent existence.

Do you "need" it in terms of a full understanding of what the aim of the Control Cards is, now, and any time anyone needs to look at them, or to copy them to carry out some other task? I'd say Yes.

I don't think it'll make any difference to the resources used (the truncation will happen with the BUILD, rather than ever-so-slightly later, that's all). It is extra typing, but not much.

For spending that extra on it, for doing the complete job, you'll save time every time someone looks at the Control Cards, copies them for another task, looks at those copied cards, or copies them again.

For me, a "no brainer".

Specifying all outputs as DISP=OLD is "interesting". What are the benefits seen to be behind that idea?
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose

PostPosted: Tue May 21, 2013 4:22 am
Reply with quote

abhijit.nayak01 wrote:
Now my only question is even if the output datset is old and it has lrecl=250 do we need
Code:
INREC BUILD=(1,250) 


You are directing your concerns in the wrong area. Earlier you are were doing 2 sorts and reading tons of unnecessary records. Instead of correcting them and learning to optimize your job, you are looking at the most mundane issue about the LRECL truncation.

You need to spend more time understanding the job or follow directions than truncation of data.

As bill pointed I would love to know the benefits of using DISP=OLD. This is not IMS checkpoint restart job, so why would any one have DISP=OLD? Lets say for some reason the joinkeys job returned an RC=16 with some kind of error, but the job doesn't abend(The default setting for DFSORT is not to abend but return an RC=16). Since you had DISP=OLD, you would get the data from earlier run? Is that really how you process the data?
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Tue May 21, 2013 9:33 pm
Reply with quote

Hello,

Possibly, there were problems in allocating output datasets in the past and this method was chosen to make sure the dataset was "there". Also, possibly the problem of getting a duplicate when the attempt to allocate is made. Poor job of housekeeping.

Far easier to do this than actually manage the dasd and the processes that allocate datasets . . .
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Tue May 21, 2013 11:57 pm
Reply with quote

Actually from the callout prospective we use DISP=OLD as the dataset is created in one cycle job which gets refreshed everyday after the batch run. We keep it DISP=OLD because if the job abends and when we ask the operator to rerun then we should not face the JCL error for dataset already present.
After every step we check the RC and if the RC is expected then we proceed to next step else abend the job.

But for the above scenario it is my mistake while testing. The input dataset in production is a GDG version and the output dataset is a temp and then the temp dataset goes to next step for creation of report. So, when I started testing I converted it to normal dataset and did the testing and I forgot and argued for LREC=250. Sorry for that. It was my mistake for misunderstanding. If everybody agrees then we can close this topic.

Code:
DSN=&&MARKOFF,DISP=(NEW,PASS),         
DCB=(RECFM=FB,LRECL=250),             
SPACE=(CYL,(2,5)),UNIT=SYSDA           
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Wed May 22, 2013 12:08 am
Reply with quote

Hello,

Quote:
when we ask the operator to rerun then we should not face the JCL error for dataset already present.

Exactly a situation that demonstrates poor process design.

Well-implemented batch jobs ALWAYS cleanup before and/or after they run. There would be NO WAY to have problems if these jobs were properly implemented.
Back to top
View user's profile Send private message
Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1788
Location: Bloomington, IL

PostPosted: Wed May 22, 2013 12:20 am
Reply with quote

dick scherrer wrote:
Quote:
when we ask the operator to rerun then we should not face the JCL error for dataset already present.

Exactly a situation that demonstrates poor process design.

Or lack of knowledge; I'd be willing to bet a decent sum that Abhijit-kun is misusing "abend" to mean "the job doesn't run entirely as expected".
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed May 22, 2013 12:28 am
Reply with quote

I've never heard of doing it that way :-)

That's definitely a "not fixing the brakes but making the horn louder".

I don't suppose you use RLSE? Client overpaying for DASD? Or have to have secondary allocation taking on the role of primary, so overallocation during the run, no "warning" of unexpected extensions in file sizes, blah, blah.

I'd suggest getting it fixed before the client finds out. Unless it was their idea in the first place, in which case give me their contact details, please, as I've got a bridge I'd like to sell to them :-)
Back to top
View user's profile Send private message
abhijit.nayak01

Active User


Joined: 22 Mar 2009
Posts: 161
Location: South Africa

PostPosted: Wed May 22, 2013 1:11 am
Reply with quote

Well the processes are already there from past. Which I can't change but I will keep in mind whenever I will put my changes.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 


Similar Topics
Topic Forum Replies
No new posts Compare only first records of the fil... SYNCSORT 7
No new posts Pulling a fixed number of records fro... DB2 2
No new posts DROP & ALTER PARTITION-PBR DB2 0
No new posts Join multiple records using splice DFSORT/ICETOOL 5
No new posts EZT program to build a flat file with... All Other Mainframe Topics 9
Search our Forums:

Back to Top