|
View previous topic :: View next topic
|
| Author |
Message |
dp33770
New User
Joined: 04 Jul 2007
Posts: 91
Location: Hyderabad
|
|
|
|
Hi All,
I have two input file each consists of huge no of records.
I need to check every record of File 1 with entire record of File 2.
BUt I cant use File OPEN and CLOSE these many number of times, which is not advisable.
Can anyone tell me if you have any simple solution for this. |
|
| Back to top |
|
 |
PeD
Active User
Joined: 26 Nov 2005
Posts: 459
Location: Belgium
|
|
|
|
| Sort the two files before than use an utility to compare the files ( DFSORT, SYNCSORT,SuperC, EComp, Comparex, ... ) |
|
| Back to top |
|
|
dp33770
New User
Joined: 04 Jul 2007
Posts: 91
Location: Hyderabad
|
|
|
|
| Its not just compare, I use each record of File1 with every rec of File2 for some more internal processing. |
|
| Back to top |
|
|
PeD
Active User
Joined: 26 Nov 2005
Posts: 459
Location: Belgium
|
|
|
|
I don't know what you mean by "huge", but if every record of file 2 is concerned by every record of file 1, what a strange process.
I suspect a poor design. Of course maybe you don't control the data to be processed.
Please ,don't jump too fast into that excuse 
Can you describe more? |
|
| Back to top |
|
|
Robert Sample
Global Moderator
Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA
|
|
|
|
| Quote: |
I have two input file each consists of huge no of records.
I need to check every record of File 1 with entire record of File 2.
BUt I cant use File OPEN and CLOSE these many number of times, which is not advisable.
Can anyone tell me if you have any simple solution for this.
|
Depending upon your definition of "huge", you have three choices:
1) Build a table in memory of all records of file 2 and use that table as you read file 1.
2) Use OPEN and CLOSE however many times are needed -- which will really, really, slow down processing and potentially cause system problems.
3) Redesign the process to not require this strange sort of processing. |
|
| Back to top |
|
|
mmwife
Super Moderator
Joined: 30 May 2003
Posts: 1592
|
|
|
|
Hi DeeP,
It might help to give a few examples of the process using "small" files (say 10 recs in one; 12 in the other).
Also provide some other info, e.g. are they both sorted on the same key; is it a 1 to 1 relationship?; many to many; many to many equal numbers of each? etc. |
|
| Back to top |
|
|
Terry Heinze
JCL Moderator
Joined: 14 Jul 2008
Posts: 1249
Location: Richfield, MN, USA
|
|
|
|
| Your description is still a bit vague. Is this a typical "transaction file updating master file" type of task? |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
| Quote: |
| each consists of huge no of records. |
If each file contains 100 million records, the number of records to be compared will be 10,000,000,000,000,000. . . . There is probably not enough mainframe available to service this . . .
As requested, post a small set of the 2 files and explain the processing requirement. Then maybe someone will have a suggestion. |
|
| Back to top |
|
|
dp33770
New User
Joined: 04 Jul 2007
Posts: 91
Location: Hyderabad
|
|
|
|
First of all I am very sorry for not properly defining my question in appropriate way.
As suggested by you all below is a sample example.
I have a master rec which has suppose the below fields
Master File
------------
Unique_Master_key
Master_data
Ex:
0001 AAAAAA
0002 BBBBBB
0003 CCCCCC
Transaction File
----------------
Transaction_No
Transaction_Data
T001 aaaa
T002 bbbb
T003 cccc
T004 dddd
T005 eeee
Now for each master key and each Transaction no combination I will look for some other field in a table , for ex.
Select * from a table
where Master_key = 0001
and transaction_no = T001
Select * from a table
where Master_key = 0001
and transaction_no = T002
Select * from a table
where Master_key = 0001
and transaction_no = T003
..
..
..
Select * from a table
where Master_key = 0002
and transaction_no = T001
Select * from a table
where Master_key = 0002
and transaction_no = T001
and so on .. |
|
| Back to top |
|
|
PeD
Active User
Joined: 26 Nov 2005
Posts: 459
Location: Belgium
|
|
|
|
... ???... strange process !
For my info, which kind of process will be triggered by the result of each select.
Too much matching data has no meaning : if anything points on anything, they cannot be specific process. |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
| Quote: |
| Now for each master key and each Transaction no combination I will look for some other field in a table |
I believe not. . . As i mentioned earlier, if the files are "huge", the process will never complete. Also, you mention "some other field" but there is none in your reply.
Your first post mentioned files, but your "sample code" is sql (which is not used with sequential files).
You need to better explain the requirement. What will be done when there is a "match"? What is to be the "output" from this process? |
|
| Back to top |
|
|
|
|
