|
View previous topic :: View next topic
|
| Author |
Message |
Ayyappa1
New User
Joined: 25 Jun 2016
Posts: 7
Location: India
|
|
|
|
Hello All,
I receive multiple generations to my GDG dataset (say TESTZZ2.ACTUAL.TEST this is a GDG Base). The number of files received may vary day to day. It can be either 0 files or 3 files or files or 2 files, etc. No specified pattern/rule for no of files.
I have a Master dataset with 30 Generations (say TESTZZ2.MASTER.DATA - GDG Base). Now my requirement is I need to compare each generation of TESTZZ2.ACTUAL.TEST with each generation of TESTZZ2.MASTER.DATA. If there is exact match of the file content, TESTZZ2.ACTUAL.TEST generation file has to be deleted. If not matched, should be left as it is.
Any suggestions on how to achieve it would be helpful.
Note : After processing the TESTZZ2.ACTUAL.TEST by other Job(say DAILPROC), an EMPTY file with next generation number will be created. Say for example, I have received 5 files today. After processing by DAILPROC, an empty generation TESTZZ2.ACTUAL.TEST.G0006V00 will be created. |
|
| Back to top |
|
 |
expat
Global Moderator
Joined: 14 Mar 2007
Posts: 8796
Location: Welsh Wales
|
|
|
|
What on earth is the logic behind this ?
Are you trying to find out if all generations have been processed ?
Is each generation an accumulation of data until a NULL is created after processing. Or is it an individual entity.
You really do need to explain the situation in far greater detail than you have. |
|
| Back to top |
|
|
Ayyappa1
New User
Joined: 25 Jun 2016
Posts: 7
Location: India
|
|
|
|
| expat wrote: |
What on earth is the logic behind this ?
Are you trying to find out if all generations have been processed ?
Is each generation an accumulation of data until a NULL is created after processing. Or is it an individual entity.
You really do need to explain the situation in far greater detail than you have. |
To explain in brief, the files are placed by Vendor. Due to some issue on Vendor end, they are placing duplicates. So I am working on to remove duplicates and process only actuals. Hope this clarifies.
Processing will be done on the actuals. As of now, this is being manually done. Due to excess of files, I am trying to automate it. Need some suggestions very seriously.
Just to add further information. After processing, all the generations will be deleted and an empty generation will be created with next generation number. |
|
| Back to top |
|
|
Nic Clouston
Global Moderator
Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK
|
|
|
|
| Do the data sets (nthey are not 'files') have headers that identify them uniquely? If so simply record them as they are read by DAILPROC and skip them if the header has already been processed. |
|
| Back to top |
|
|
Ayyappa1
New User
Joined: 25 Jun 2016
Posts: 7
Location: India
|
|
|
|
| Nic Clouston wrote: |
| Do the data sets (nthey are not 'files') have headers that identify them uniquely? If so simply record them as they are read by DAILPROC and skip them if the header has already been processed. |
Data doesn't have any header or trailer information  |
|
| Back to top |
|
|
expat
Global Moderator
Joined: 14 Mar 2007
Posts: 8796
Location: Welsh Wales
|
|
|
|
So basically the application DAILPROC has been badly designed and does not contain the logic to deal with duplicate records.
Have you considered doing a sort on all of the files to be used as input and then removing any duplicate records ?
Simple but effective  |
|
| Back to top |
|
|
Ayyappa1
New User
Joined: 25 Jun 2016
Posts: 7
Location: India
|
|
|
|
| DAILPROC has a different logic. It copies the file to different GDG's based on content available in it. So Sorting and verifying will not work as expected. |
|
| Back to top |
|
|
Nic Clouston
Global Moderator
Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK
|
|
|
|
| Do a LISTC of the daily generations and interpret it to create the JCL required to do the comparisons and reporting. |
|
| Back to top |
|
|
Ayyappa1
New User
Joined: 25 Jun 2016
Posts: 7
Location: India
|
|
|
|
| Nic Clouston wrote: |
| Do a LISTC of the daily generations and interpret it to create the JCL required to do the comparisons and reporting. |
well. I didn't get that, can you elaborate a bit more in detail? Appreciate your help on this. |
|
| Back to top |
|
|
Nic Clouston
Global Moderator
Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK
|
|
|
|
| Use IDCAMS to generate a list of all the data sets associated with the ACTUAL.TEST base. Process that list to extract the data set names and generate the required JCL for each comparison. If you have 5 generations of ACTUAL.TEST and you have to compare each one against each data set of MASTER.DATA then you will have to generate 150 compares. By suitable use of condition codes you could probably bypass further compares of the current ACTUAL.TEST generation to avoid excessive execution times. |
|
| Back to top |
|
|
Ayyappa1
New User
Joined: 25 Jun 2016
Posts: 7
Location: India
|
|
|
|
| Nic Clouston wrote: |
| Use IDCAMS to generate a list of all the data sets associated with the ACTUAL.TEST base. Process that list to extract the data set names and generate the required JCL for each comparison. If you have 5 generations of ACTUAL.TEST and you have to compare each one against each data set of MASTER.DATA then you will have to generate 150 compares. By suitable use of condition codes you could probably bypass further compares of the current ACTUAL.TEST generation to avoid excessive execution times. |
Thats a good one. But I have one more issue here. I don't have the standard limit(constant) number of files received daily. It may be 20 generations today, next day it can be 5, next day it can be any number. So the problem is identifying no.of generations available in runtime(dynamically).
Any suggestions for this situation? |
|
| Back to top |
|
|
Nic Clouston
Global Moderator
Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK
|
|
|
|
| Does your daily generation start at G0001V00 or does it continue from the empty one created by DAILPROC? If the former then there is no problem - you generate for each data set until you have processed them all. If the latter, you can either interrogate the date created field of the IDCAMS listing and compare to the run date or you can keep a record of the last generation processed (compared). |
|
| Back to top |
|
|
Ayyappa1
New User
Joined: 25 Jun 2016
Posts: 7
Location: India
|
|
|
|
| Nic Clouston wrote: |
| Does your daily generation start at G0001V00 or does it continue from the empty one created by DAILPROC? If the former then there is no problem - you generate for each data set until you have processed them all. If the latter, you can either interrogate the date created field of the IDCAMS listing and compare to the run date or you can keep a record of the last generation processed (compared). |
It doesn't start from G0001V00. It starts from Empty generation created by DAILPROC. |
|
| Back to top |
|
|
Willy Jensen
Active Member
Joined: 01 Sep 2015
Posts: 740
Location: Denmark
|
|
|
|
| So you will need to do a LISTCAT LVL(xxx) and post-process the output to get the actual names. |
|
| Back to top |
|
|
|
|
