Spliting a file into different files using SORT

abin · Active User Joined: 14 Aug 2006 Posts: 198

Hi All,

Please provide some light this problem.

I have a file of length 5000.

Its layout is of format
01 ws-layout.
05 ws-key1 pic x(20)
05 FILLER pic x(20)
05 ws-key2 pic x(10)
05 FILLEr pic x(4950).

The file contains more than some 35 million records.

I have to spilt this file into 10 different files. But none of this file should have the key values overlapped.

ie if file1 have a record with key1 nnone of other files have a record with the same key.

Please let me know if there needs some more clarrification.

Thanks,
Abin

IQofaGerbil · Posted: Mon Mar 12, 2007 7:07 pm

More clarification? Yes please!

Examples of inputs and expected outputs would certainly help.

abin · Active User Joined: 14 Aug 2006 Posts: 198

Hi,

Thanks for replying.

Input wuld look like
1111111111111111111SOMEDATA1 1111111111
1111111111111111111SOMEDATA2 1111111111
2222222222222222222SOMEDATA3 2222222222
3333333333333333333SOMEDATA4 3333333333
3333333333333333333SOMEDATA5 3333333333
3333333333333333333SOMEDATA6 3333333333
4444444444444444444SOMEDATA7 4444444444
5555555555555555555SOMEDATA8 5555555555
6666666666666666666SOMEDATA9 6666666666
6666666666666666666SOMEDATA10 6666666666
9999999999999999999SOMEDATA11 9999999999

PS: Input file is in sorted order

I want to split this file into two part
Now when I split first file shuld contain
1111111111111111111SOMEDATA1 1111111111
1111111111111111111SOMEDATA2 1111111111
2222222222222222222SOMEDATA3 2222222222
3333333333333333333SOMEDATA4 3333333333
3333333333333333333SOMEDATA5 3333333333
3333333333333333333SOMEDATA6 3333333333
and second file shuld contain
4444444444444444444SOMEDATA7 4444444444
5555555555555555555SOMEDATA8 5555555555
6666666666666666666SOMEDATA9 6666666666
6666666666666666666SOMEDATA10 6666666666
9999999999999999999SOMEDATA11 9999999999

PS: This is only a sample data. Actuall data contains more than 35 million records and I want to split into more than 10 different files all containing neraly equal number of records.

Thanks,
Abin

santhunaveen · New User Joined: 22 Sep 2006 Posts: 33

Hi,

We can split the files using OUTFIL option. Check below example.

Code:

//***********************************************************
//SPLITFLS EXEC PGM=SORT
//***********************************************************
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SORTIN DD DSN=FILE1......,DISP=SHR
//SORTOF01 DD DSN=OUTPUTFILE1.......,
// DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
// SPACE=(CYL,(1,1),RLSE),
// RECFM=FB,LRECL=20
//SORTOF02 DD DSN=OUTPUTFILE2..............,
// DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
// SPACE=(CYL,(1,1),RLSE),
// RECFM=FB,LRECL=20
//SORTOF03 DD DSN=OUTPUTFILE3..............,
// DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
// SPACE=(CYL,(1,1),RLSE),
// RECFM=FB,LRECL=20
//SORTOF04 DD DSN=OUTPUTFILE3..............,
// DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
// SPACE=(CYL,(1,1),RLSE),
// RECFM=FB,LRECL=20
????????????????// u can give as many files as u want
????????????????
????????????????
//SYSIN DD *
SORT FIELDS=COPY
OUTFIL FILES=01,ENDREC=200 // for file 1 how many records u want to split
OUTFIL FILES=02,STARTREC=201,ENDREC=400 // for file 2
OUTFIL FILES=03,STARTREC=401,ENDREC=700 // for file 3
OUTFIL FILES=03,STARTREC=401,ENDREC=700 // for file 4
...........................// for all files
..............................
................................
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//*

Please correct me if i'm wrong.......

IQofaGerbil · Posted: Tue Mar 13, 2007 4:31 pm

The actual requirement is to split the main file into several files WITHOUT the keys wrapping over.

Simply using an arbiitrary number will not satisfy this requirement.

William Thompson · Posted: Tue Mar 13, 2007 5:18 pm

abin · Active User Joined: 14 Aug 2006 Posts: 198

Hi Will,

Yes the records are sorted by the key.

If you want all 10 files to be nearly the same size, do you know how many are in each keyrange?

This I am afraid is not predictable. But we could divide the entire number by number of files needed. Say suppose Input contains 3000000 records
and I want 10 files. Then each file should contain nearly = 300000 records

William Thompson · Posted: Tue Mar 13, 2007 6:08 pm

OK, every 300,000 writes you want to wait for a key break and close that file and open a new one, right?
Easily done with a programming language but I don't know if it can be done with sort.
Is sort a requirement or do you have other resources that might be used?

abin · Active User Joined: 14 Aug 2006 Posts: 198

Hi will,

every 300,000 writes you want to wait for a key break and close that file and open a new one, right?

you are partly right. I dont want to wait for key break, since this is going to be a batch job.

You are right we could to do it well using a programming language. But if could it with SORT it would be great.

sril.krishy · Posted: Tue Mar 13, 2007 6:36 pm

Hi,
I think you can create the dynamic control cards to devide the file into 10 files.

Anyway let's wait for Frank's responce.He might throw good idea.

Thank you
Krishy

santhunaveen · New User Joined: 22 Sep 2006 Posts: 33

Hi IQofaGerbil,

"The actual requirement is to split the main file into several files WITHOUT the keys wrapping over. "

If the file is already sorted means....then where is the question of keys wrapping over......................

IQofaGerbil · Posted: Tue Mar 13, 2007 9:20 pm

Hi back santhunaveen

Well, from the expected output

abin · Active User Joined: 14 Aug 2006 Posts: 198

Hi Gerbil,

what you said is correct.

Frank Yaeger · Posted: Tue Mar 13, 2007 10:59 pm

This is very tricky and takes one merge pass and several copy passes, but here's a DFSORT/ICETOOL job that I believe will do what you want: