|
View previous topic :: View next topic
|
| Author |
Message |
Gururajan Balaji
New User
Joined: 10 Jan 2011
Posts: 6
Location: Chennai
|
|
|
|
Hi All,
I Have to compare 2 files on (1,9 bytes) and get matching records from file2.
First file lrecl=10,recfm=fb -> Has no dups on key (May contain 1 million records)
second file lrecl=274,recfm=fb --> has dups on key (may contain 500 million records or more)
both the files are sorted on key.
I use the below sort card for comparison.
| Code: |
JOINKEYS FILE=F1,FIELDS=(1,9,A),SORTED
JOINKEYS FILE=F2,FIELDS=(4,9,A),SORTED
REFORMAT FIELDS=(F2:1,274)
SORT FIELDS=(4,9,CH,A)
|
The problem is file2 is a very huge file (may contain 500 million records) and hence the job runs for a very long time.
Is there a way to optimize this sort? (Like allocating maximum size for buffer).
Few of my colleagues are suggesting me to go for COBOL instead of SORT for such a huge file comparison.
Could some one tell me which method could be optimal? |
|
| Back to top |
|
 |
daveporcelan
Active Member
Joined: 01 Dec 2006
Posts: 792
Location: Pennsylvania
|
|
|
|
If file two is already sorted on the key, then why not use
to do a COPY instead of a SORT?
I would think this in itself will improve performance.
I do not think Cobol would work nearly as well. |
|
| Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA
|
|
|
|
In addition to doing a COPY instead of a SORT (as suggested by Dave), I would also try reversing the files so the file with dups and more records is used as F1 to see if that performs better:
| Code: |
...
//FB274 DD DSN=... input file2 (FB/274)
//FB10 DD DSN=... input file1 (FB/10)
...
JOINKEYS F1=FB274,FIELDS=(4,9,A),SORTED
JOINKEYS F2=FB10,FIELDS=(1,9,A),SORTED
REFORMAT FIELDS=(F1:1,274)
SORT FIELDS=COPY
|
|
|
| Back to top |
|
|
Gururajan Balaji
New User
Joined: 10 Jan 2011
Posts: 6
Location: Chennai
|
|
|
|
Thanks Dave & Frank.
Will there be an Improvement in performance if I use PARM='CORE=MAX'? |
|
| Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA
|
|
|
|
| CORE=MAX is equivalent to MAINSIZE=MAX which is the shipped default for DFSORT so you won't see any difference unless your site has changed the shipped default. Since I don't know what your site defaults are, I can't tell you if CORE=MAX will have any effect. |
|
| Back to top |
|
|
David Eisenberg
New User
Joined: 15 Nov 2007
Posts: 39
Location: New York
|
|
|
|
>I would also try reversing the files so the file with dups and more records is used as F1 to see if that performs better<
Frank,
Can you tell us a little more about what prompted this suggestion? Are there generally-applicable performance considerations when choosing which file should be F1 and which should be F2 in a JOIN when one (or both) of the files is very large?
Thanks,
David |
|
| Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005
Posts: 7129
Location: San Jose, CA
|
|
|
|
| The only thing I can say is that using F1 for the file that has "more" duplicates may result in more efficient processing. I can't really be more specific than that as it depends on a lot of factors. The only way to know for sure is to try it both ways. |
|
| Back to top |
|
|
David Eisenberg
New User
Joined: 15 Nov 2007
Posts: 39
Location: New York
|
|
|
|
Frank,
That's helpful; thanks.
David |
|
| Back to top |
|
|
|
|
