View previous topic :: View next topic
|
Author |
Message |
rocky_balboa
New User
Joined: 24 Mar 2010 Posts: 61 Location: Cape of Good Hope
|
|
|
|
Hi
We have a batch system where we are onboarding data from different other systems. The steps that we have are :
1. A specific program(s) for the given source system for data validation and transformation in a standard layout.
2. Process the records in the standard layout. Viz. processing+inserting into tables for subsequent web display and reporting
Till now we had specific programs for each source system and each source system had its own vagaries. Some send a single file others multiple files, some send true delta others send partial delta.
For clarity, partial delta means that the source system just knows that something has changed in it but does not know what the change is. We find out the change in our process.
The last system which we on boarded had partial delta and multiple files. I designed the process using VSAM as a master files. I had considered DB2 as well but found VSAM better in performance for our requirements. The other factors were flexibility in design/coding as column addition/deletion would have required DBAs who have their own ETC.
Given this background, I want to create a generic program(s) (DB2/VSAM/PS) to onboard any systems’ data. The assumptions are:
• There would be only one i/p file
• The i/p data can be true delta/partial delta full load( This would be in case if the i/p record count is not significant)
Any inputs/suggestions would help me come up with a better design and avoiding pitfalls would be great. Let me know in case you need any clarification.
Thanks
Rocky |
|
Back to top |
|
|
sureshpathi10
Active User
Joined: 03 May 2010 Posts: 154 Location: Kuala Lumpur
|
|
|
|
I guess, you've figured out everything.
Considering your given scenario, I've a single suggestion that might increase your performance.
For the 1st Step
Quote: |
1. A specific program(s) for the given source system for data validation and transformation in a standard layout. |
You can use SORT utility instead of a COBOL program (if the validation can be done in SORT). |
|
Back to top |
|
|
rocky_balboa
New User
Joined: 24 Mar 2010 Posts: 61 Location: Cape of Good Hope
|
|
|
|
Additional information :
The data load happens in parts
1) The historical data load where in all the eligible data is on boarded
2) Subsequently daily delta feeds are received to update data for relevant records.
All this happens through the programs of the first stage.( Validation and transformation.
It is also assumed that each record also has an unique identifier for historical as well as delta updates.
I am not concerned about validation part here but the delta calculation in terms of efficiency and flexiblity for both the i/p file layout and the program design. |
|
Back to top |
|
|
|