View previous topic :: View next topic
|
Author |
Message |
himanshupant
New User
Joined: 21 Mar 2007 Posts: 46 Location: India
|
|
|
|
Hello
I have an input csv file containing a huge number of records ( say 4,00,000).
Now as MS Excel 2003 can not have more than 65,536 rows in a work sheet. I need to break this huge csv file into smaller csv files based on record count.. Say first 50,000 records in file 1, next 50,000 in file 2 , so on and so forth.
Is there any way to do this dynamically in ICETOOL / DFSORT ? A point to be noted is that I am not aware of the number of input records so the number of output files is not known to me before hand.
Furthermore if my input file contains the header as
name,age,address,sex,card-no
is there any way I can keep the same as header for each of the splitted file ? |
|
Back to top |
|
|
smijoss
Active User
Joined: 30 Aug 2007 Posts: 114 Location: pune
|
|
|
|
have you checked SMART DFSORT TRICKS.pdf it has 5 methods to split a file.
regarding the header :
you can have 2 sort steps :
1) to read 1st line and create a temp sort card with header and split options
2) use the sort card created above to split file |
|
Back to top |
|
|
sqlcode1
Active Member
Joined: 08 Apr 2010 Posts: 577 Location: USA
|
|
|
|
smijoss,
OP doesn't know the count of records and hence total number of output files required. He is asking for a jcl which dynamically allocates output files which has fixed number of records (50000).
Quote: |
A point to be noted is that I am not aware of the number of input records so the number of output files is not known to me before hand. |
himanshupant,
What is the LRECL of the input file?
Thanks, |
|
Back to top |
|
|
himanshupant
New User
Joined: 21 Mar 2007 Posts: 46 Location: India
|
|
|
|
sqlcode1 wrote: |
smijoss,
OP doesn't know the count of records and hence total number of output files required. He is asking for a jcl which dynamically allocates output files which has fixed number of records (50000).
Quote: |
A point to be noted is that I am not aware of the number of input records so the number of output files is not known to me before hand. |
himanshupant,
What is the LRECL of the input file?
Thanks, |
The LRECL can be taken as 100 bytes |
|
Back to top |
|
|
Frank Yaeger
DFSORT Developer
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
|
|
|
|
See the "Split a file to n output files dynamically" Smart DFSORT Trick at:
www.ibm.com/support/docview.wss?rs=114&uid=isg3T7000094
Note that you must supply the maximum possible number of output DD statements in the JCL.
If you really want to set up the output DDs dynamically, you would have to create a job for the internal reader.
If you need more specific help, show an example of the records in your input file and what you expect for output. Give the RECFM and LRECL of the input file. |
|
Back to top |
|
|
himanshupant
New User
Joined: 21 Mar 2007 Posts: 46 Location: India
|
|
|
|
Frank Yaeger wrote: |
See the "Split a file to n output files dynamically" Smart DFSORT Trick at:
www.ibm.com/support/docview.wss?rs=114&uid=isg3T7000094
Note that you must supply the maximum possible number of output DD statements in the JCL.
If you really want to set up the output DDs dynamically, you would have to create a job for the internal reader.
If you need more specific help, show an example of the records in your input file and what you expect for output. Give the RECFM and LRECL of the input file. |
Thanks a ton Frank - The document really looks helpful !!! |
|
Back to top |
|
|
smijoss
Active User
Joined: 30 Aug 2007 Posts: 114 Location: pune
|
|
|
|
hi sqlcode1,
never heard of JCL allocating dynamic files .. so didnt get into that ;) |
|
Back to top |
|
|
sqlcode1
Active Member
Joined: 08 Apr 2010 Posts: 577 Location: USA
|
|
|
|
smijoss,
Quote: |
He is asking for a jcl which dynamically allocates output files which has fixed number of records (50000). |
Its not dynamic allocation through JCL but as rather setting up JCL DD statements (and other required JCL statements as well) dynamically through JOB and later submitting the job through Internal Reader.
Frank Yaeger wrote: |
If you really want to set up the output DDs dynamically, you would have to create a job for the internal reader.
|
Thanks, |
|
Back to top |
|
|
|