|
View previous topic :: View next topic
|
| Author |
Message |
shilpa.khaire
New User
Joined: 21 Jul 2006
Posts: 39
Location: US
|
|
|
|
Hi All!
I have one file which I want to split based on date. File 1 and file2.
I again want to split File2 based on some other field(not date) .
Can this be done in a single step? |
|
| Back to top |
|
 |
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
Suggest you post a few sample records from both files and the output you want from these samples.
Why would a "single step" be an issue?
You could always write a bit of really simple code that had 2 processes (1 for each input file). |
|
| Back to top |
|
|
shilpa.khaire
New User
Joined: 21 Jul 2006
Posts: 39
Location: US
|
|
|
|
I agree to you that it better to break up the processes.
But I want to check if combining into a single step will reduce the cPU time.
e.g.
Input file - Split on date
2005 A
2006 B
2010 A
2011 B
File 1 - Records prior to 2009
2005 A
2006 B
File 2 - Records after 2009
2010 A
2011 B
File 2 - Split based on Second field
My final output will be:
File 3
2010 A
File 4
2011 B
Thanks! |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
So, if i understand, you want to split file1 into 3 files? Is there a need for the file2 data to be in that file or is it only needed for the second split?
It will help if you post the "rules" for the second split. |
|
| Back to top |
|
|
shilpa.khaire
New User
Joined: 21 Jul 2006
Posts: 39
Location: US
|
|
|
|
I dont want the file 1 to be exactly into 3 files. Rules for the second split are - Suppose File 2 is a system-id whcih has values A,B,C and D. In that case, I want the file 2 to be split up by the diffreent values in System-id. So in this case we will have 4 files coming out of the second split.
File 1 - Records prior to 2009
2005 A
2006 B
File 2 - Records after 2009
2010 A
2011 B
2011 A
2012 C
2012 D
File 2 - Split based on Second field
My final output will be:
File 3
2010 A
2011 A
File 4
2011 B
File 5
2012 C
File 4
2012 D
Answer to the 2'nd question is I need file2 seprately as a O/P of the first split. And then I need the outputs of the second split separately too.
Basically my idea is to achieve some saving in the run time by merging the process.
Thanks for looking into this! |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
| Quote: |
| Basically my idea is to achieve some saving in the run time by merging the process. |
The only way to save some time is to process less. . .
One way to do this is to read all of the data only once and write all of the output during that read. If the data is read/written/read again/written again as proposed it will take about the same time in one step or 2.
You will need to think thru how the "second split" will be managed. Processes that can create an unspecified number of outputs are often a support/maintenance nightmare.
Something simple that can be looked into is improving the block size(s) of the files. |
|
| Back to top |
|
|
shilpa.khaire
New User
Joined: 21 Jul 2006
Posts: 39
Location: US
|
|
|
|
Thats the reason why I was thinking of one SORT step...and split it up in one read of the input file.
Can we have a single SORT for this? |
|
| Back to top |
|
|
shilpa.khaire
New User
Joined: 21 Jul 2006
Posts: 39
Location: US
|
|
|
|
| dick scherrer wrote: |
Hello,
Something simple that can be looked into is improving the block size(s) of the files. |
The Block size is all 0. If we read the input file once to create the splits by date as well as second field , will it not be faster, since it is just one pass through the input file.
The elapsed time for all these steps is around 3 to 4 hours, and I know its not due to data contention. What could be teh other reason for elapsed time being so high?
Thanks! |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
| Quote: |
| and I know its not due to data contention. What could be teh other reason for elapsed time being so high? |
How could it be known that there is not a contention issue? Contention exists far beyond only the files used in the process. . .
The main reason for the elapsed time is probably due to the volume of data being processed. How many records are there in the original input file? What is their lrecl? |
|
| Back to top |
|
|
shilpa.khaire
New User
Joined: 21 Jul 2006
Posts: 39
Location: US
|
|
|
|
We found that the elapsed time is high due to files getting migrated.
We cannot move the job creating this file but we can run an IEFBR just before our job(in paralllel with the job which triggers our job).
Is there any solution to this?
Thanks! |
|
| Back to top |
|
|
|
|
