View previous topic :: View next topic
|
Author |
Message |
shaktiprasad
New User
Joined: 20 Aug 2006 Posts: 34 Location: chennai
|
|
|
|
Hi ,
Just a clarification. When we do some file comparison based on some keys. If we have billions of records then which will be better in performnce ICETOOL or Cobol ?
I know ICETOOL will easier to code but performnce wise need to decide also. I will be using SPLICE as I need to overwrite few fields based on some condition.
Thanks
Shakti |
|
Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
|
|
|
|
Hello,
Suggest you look at DFSORT's JOINKEYS function. |
|
Back to top |
|
|
sqlcode1
Active Member
Joined: 08 Apr 2010 Posts: 577 Location: USA
|
|
|
|
shaktiprasad,
Other than looking at JOINKEYS, try with few million records first and see which one is more efficient. You should get answer by yourself.
If you do, cobol file comparison,wouldn't Cobol need both files "pre-sorted" on key fields?
Thanks, |
|
Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
|
|
|
|
Hello,
No matter how you do this, you need to make sure the input is sorted first. . .
If the volume is too great, the inputs can be split by ranges and then compared. Depending on just how much data there is (lrecl * nbr-recs) you may not be able to find enough work dasd. . . |
|
Back to top |
|
|
PeterHolland
Global Moderator
Joined: 27 Oct 2009 Posts: 2481 Location: Netherlands, Amstelveen
|
|
|
|
Quote: |
billions of records
|
Really? |
|
Back to top |
|
|
Bill Woodger
Moderator Emeritus
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
|
|
|
|
I'd say that for the I/O DFSORT/ICETOOL for a few billion records would knock Cobol into a cocked hat. Hare vs Tortoise, without any danger of the Hare going to sleep.
If you look at the Cobol manual for compiler option FASTSRT (or similar name), you get the idea if you want the faster processing of the files, let Sort do the IO.
For such volumes, you really want to get it right first time, and be nice and stable, to avoid lost time due to silly failures.
If you give us the information about the rest of the processing, I'm sure you can get some very good advice.
As sqlcode1 has said, get yourself some decent-sized, and representative, test files. For this task, performance is going to be important to you.
If some of the processing is too complex, there is always the option of "Sort Exit"s, modules which you write, can be in Cobol, and are called by the sort itself (saving Cobol reading the file(s) again). |
|
Back to top |
|
|
sqlcode1
Active Member
Joined: 08 Apr 2010 Posts: 577 Location: USA
|
|
|
|
shaktiprasad,
JOINKEYS has been available for DFSort for quite a some time now. Read JOINKEYS for DFSort here
Quote: |
No matter how you do this, you need to make sure the input is sorted first. . . |
Dick,
DFSORT can sort and join the files in the same step. Input files doesn't have to pre-sorted as an input to DFSort JOINKEYS.
With Cobol, unless OP is using internal sort for this huge volume, he would need separate steps to sort both the input files on key fields first and then feed these 2 files in cobol, whereas with DFSort Joinkey, he doesn't have to do this in separate steps as JOINKEYS would sort the keys and match the files in a single step.
However, if I am to process files with "billions of records" and using JOINKEYS for match, I would go with separate sort steps each files and follow it up with SORTED,NOSEQCK in JOINKEYS.
Thanks, |
|
Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
|
|
|
|
Hi Dick,
I was somewhat surprised that the organization would even allow someone with so little experience to implement this process that has the potential to dim the lights in the data center. . .
My personal approach to monsters like this is to break them into more managable pieces - both for my sanity and to be a considerate corporate citizen on the system. . .
d |
|
Back to top |
|
|
|