Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

Stop reading input file based on a condition?

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL
View previous topic :: :: View next topic  
Author Message
David Sde

New User


Joined: 24 Apr 2011
Posts: 23
Location: USA

PostPosted: Fri Feb 22, 2013 1:10 am    Post subject: Stop reading input file based on a condition?
Reply with quote

I hope someone can help...

I have a very large input file which is already sorted on a given key. I'd like to use DFSORT with OPTION COPY to copy (and modify) some of those records, and write them to another dataset. I gather that I can stop reading the input file after a given number of records, but is there a way that I can stop reading the input file based on a condition? E.g., if the file is known to be presorted alphabetically on the first byte of each record, and I know that I'm not interested in any record where the first byte is > C'D', is there a way I can tell DFSORT to stop reading when it sees a key >= C'E' ? (If it matters, these are variable-length records.)

My goal, of course, is to speed up the process by not bothering to read records that I know will be discarded anyway.

Thanks so much,

David
Back to top
View user's profile Send private message

Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1738
Location: Bloomington, IL

PostPosted: Fri Feb 22, 2013 1:33 am    Post subject:
Reply with quote

You should be able to use INCLUDE or OMIT to select records to be output (or not).
Back to top
View user's profile Send private message
Binop B

Active User


Joined: 18 Jun 2009
Posts: 407
Location: Nashville, TN

PostPosted: Fri Feb 22, 2013 1:33 am    Post subject:
Reply with quote

Hi David,

If, as you say, the file is sorted... why dont you just do a OMIT option with condition as OMIT if key > '<last value of concern>'
Back to top
View user's profile Send private message
David Sde

New User


Joined: 24 Apr 2011
Posts: 23
Location: USA

PostPosted: Fri Feb 22, 2013 3:04 am    Post subject: Reply to: Stop reading input file based on a condition?
Reply with quote

Yes, I can certainly use OMIT to get the output file I want; what I'm wondering is whether I can speed the process up, by not reading unwanted input records in the first place. Suppose my input file contains 500 million records. If I know that the file is presorted, and if I could tell DFSORT to stop the COPY operation when it sees a particular key, then I might be able to get away with reading only 1 million records. So what I'm wondering is whether I can do something like that.

David
Back to top
View user's profile Send private message
Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1738
Location: Bloomington, IL

PostPosted: Fri Feb 22, 2013 3:33 am    Post subject:
Reply with quote

Ah; well, I think you'll need to write an E15 exit routine, but I think that Sri Kolusu or Mr. Woodger will be much better resources than I for such an endeavor.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7236

PostPosted: Fri Feb 22, 2013 5:33 am    Post subject: Reply to: Stop reading input file based on a condition?
Reply with quote

Mmmm.... tricky. I don't know of a way to stop on a key, except by using an EXIT which is going to insert all the records for SORTING/COPYING (ie, no SORTIN specified) in which case, for a COPY operation, why bother with an EXIT rather than just a program?

If you set the RETURN-CODE to 8 from an input EXIT, the EXIT won't be called again, but the rest of the file will be processed by DFSORT.

If you set the RETURN-CODE to 16, DFSORT will end, with an RC of 16. This, or another way to get RC=16 from a MERGE (destroy the key after the one you want as your last) would be messy ways to do it, and you'd need confirmation that any output files are valid (I've never checked).

As far as I'm aware, if you give DFSORT a SORTIN, and you don't have STOPAFT, and you don't limit the number of records on all OUTFILs then DFSORT will read the entire SORTIN.

One for Kolusu :-)
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose

PostPosted: Fri Feb 22, 2013 6:22 am    Post subject: Re: Reply to: Stop reading input file based on a condition?
Reply with quote

David Sde wrote:
Yes, I can certainly use OMIT to get the output file I want; what I'm wondering is whether I can speed the process up, by not reading unwanted input records in the first place. Suppose my input file contains 500 million records. If I know that the file is presorted, and if I could tell DFSORT to stop the COPY operation when it sees a particular key, then I might be able to get away with reading only 1 million records. So what I'm wondering is whether I can do something like that.

David


David,

There is no way to stop reading the other records just based on condition. However you can stop reading after a certain number of records are picked based on the condition.

ex:

Code:

//STEP0100 EXEC PGM=SORT           
//SYSOUT   DD SYSOUT=*             
//SORTIN   DD *                     
A                                   
B                                   
D                                   
A                                   
E                                   
A                                   
A                                   
H                                   
//SORTOUT  DD SYSOUT=*             
//SYSIN    DD *                     
  INCLUDE COND=(1,1,CH,EQ,C'A')     
  OPTION COPY,STOPAFT=2             
//*


Code:

ICE055I 0 INSERT 0, DELETE 2       
ICE054I 0 RECORDS - IN: 4, OUT: 2 


DFSORT would read only 4 records as the INCLUDE cond is executed first and we pick only 2 records.

If you don't specify the STOPAFT , then you are reading the entire file as DFSORT doesn't know which records match the condition until it read the entire file.

ex:
Code:

//STEP0150 EXEC PGM=SORT         
//SYSOUT   DD SYSOUT=*           
//SORTIN   DD *                 
A                               
A                               
A                               
A                               
A                               
D                               
E                               
H                               
//SORTOUT  DD SYSOUT=*           
//SYSIN    DD *                 
  INCLUDE COND=(1,1,CH,EQ,C'A') 
  OPTION COPY                   
//*


Code:

ICE055I 0 INSERT 0, DELETE 3       
ICE054I 0 RECORDS - IN: 8, OUT: 5   
Back to top
View user's profile Send private message
David Sde

New User


Joined: 24 Apr 2011
Posts: 23
Location: USA

PostPosted: Fri Feb 22, 2013 7:15 am    Post subject: Reply to: Stop reading input file based on a condition?
Reply with quote

Quote:
If you don't specify the STOPAFT , then you are reading the entire file as DFSORT doesn't know which records match the condition until it read the entire file.


Thank you, Kolusu... I see the issue. To do what I'm looking for, I would actually need two things: 1) the ability to specify a stop condition, and b) a way to tell DFSORT that SORTIN is presorted, so that it would reasonably know that it didn't have to read the entire file. Looks like I'll have to waste some cycles!

Thank you for your responses, everyone; I appreciate it.

David
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7236

PostPosted: Fri Feb 22, 2013 7:45 am    Post subject: Reply to: Stop reading input file based on a condition?
Reply with quote

Whether SORTIN is in sequence or not is irrelevant to a COPY operation. It is relevant to your task, of course.

Generally, if a file is already "in sequence", you'd not SORT it, so DFSORT never needs to know, for a SORT, that the data is already in sequence.

For MERGE, the "in sequence" is a necessity, so DFSORT tells you in no uncertain terms when it is not.

You can't do what you want with DFSORT unless you go for the insecure clunkiness of getting an RC=16 and DFSORT stopping dead.

So, write a program.

You'll lose out on DFSORT's superior IO performance, but gain by being able to elegantly stop where you want.

If you have 500,000,000 records, I believe you will have considerable resource savings with a random distribution of "stop keys". If all your "stop keys" are in the last 100,000,000 records, you'd have to do some comparisons of the approaches to determine the most effective.

Is the file on DASD? You could consider "splitting" it into multiples, perhaps, and have knowledge of the key ranges. When the entire file is needed, concatenate all.

The JCL/control cards to accomplish a particular extract could be generated, and either sent to the INTRDR or submitted separately for execution.
Back to top
View user's profile Send private message
Alain Benveniste

New User


Joined: 14 Feb 2005
Posts: 88

PostPosted: Sun Feb 24, 2013 5:43 pm    Post subject:
Reply with quote

It was an enhancement I talked with Frank many years ago...

Alain
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Removing Duplicates based on certain ... chandracdac DFSORT/ICETOOL 8 Fri Dec 09, 2016 4:40 am
No new posts Performing arithmetic on input field zh_lad DFSORT/ICETOOL 24 Tue Dec 06, 2016 8:04 pm
No new posts High CPU consumption Job using IAM fi... aswinir JCL & VSAM 8 Thu Dec 01, 2016 8:28 pm
No new posts Add PD field from 2nd file to PD in 1st Sushant Garje DFSORT/ICETOOL 6 Thu Dec 01, 2016 4:32 pm
No new posts File Aid to File Manager conversion murali3955 IBM Tools 4 Thu Nov 24, 2016 3:41 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us