Joined: 24 Mar 2010 Posts: 61 Location: Cape of Good Hope
We have a batch system where we are onboarding data from different other systems. The steps that we have are :
1. A specific program(s) for the given source system for data validation and transformation in a standard layout.
2. Process the records in the standard layout. Viz. processing+inserting into tables for subsequent web display and reporting
Till now we had specific programs for each source system and each source system had its own vagaries. Some send a single file others multiple files, some send true delta others send partial delta.
For clarity, partial delta means that the source system just knows that something has changed in it but does not know what the change is. We find out the change in our process.
The last system which we on boarded had partial delta and multiple files. I designed the process using VSAM as a master files. I had considered DB2 as well but found VSAM better in performance for our requirements. The other factors were flexibility in design/coding as column addition/deletion would have required DBAs who have their own ETC.
Given this background, I want to create a generic program(s) (DB2/VSAM/PS) to onboard any systems’ data. The assumptions are:
• There would be only one i/p file
• The i/p data can be true delta/partial delta full load( This would be in case if the i/p record count is not significant)
Any inputs/suggestions would help me come up with a better design and avoiding pitfalls would be great. Let me know in case you need any clarification.