DFSORT taking too long time to sort

Singaram · New User Joined: 19 Aug 2009 Posts: 19 Location: India

The following sort is taking too long time:

1. Should I change the size to MAX?
2. Should I consider increasing BUFNO for the sequential datasets
3. Should I merge all these files before issuing the sort step.

Any advice on performance will be greatly appreciated

XXSORT0005 EXEC PGM=SORT,COND=(8,LT),
XX PARM='SIZE=60M'
39 XXSYSOUT DD SYSOUT=*
40 XXSORTWK01 DD UNIT=SRTDA,SPACE=(CYL,(3000,22))
41 XXSORTWK02 DD UNIT=SRTDA,SPACE=(CYL,(3000,22))
42 XXSORTWK03 DD UNIT=SRTDA,SPACE=(CYL,(3000,22))
43 XXSORTWK04 DD UNIT=SRTDA,SPACE=(CYL,(3000,22))
44 XXSORTWK05 DD UNIT=SRTDA,SPACE=(CYL,(3000,22))
45 XXSORTWK06 DD UNIT=SRTDA,SPACE=(CYL,(3000,22))
46 XXSORTWK07 DD UNIT=SRTDA,SPACE=(CYL,(3000,22))
47 XXSORTWK08 DD UNIT=SRTDA,SPACE=(CYL,(3000,22))
48 XXSORTIN DD DSN=AMJOS.JDFDS.TRANS.EDOSCTA.DEPURADO,
XX DISP=OLD,DCB=BUFNO=12
49 XX DD DSN=HLQ.EDOSCTA.BATCH,
XX DISP=OLD,DCB=BUFNO=2
50 XX DD DSN=HLQ.MOVS.EDOCTA,
XX DISP=OLD,DCB=BUFNO=2
51 XX DD DSN=HLQ.ESTADO.CUENTA,
XX DISP=OLD,DCB=BUFNO=2
52 XX DD DSN=HLQ.EDOCTA.NPOS,
XX DISP=OLD,DCB=BUFNO=2
53 XX DD DSN=HLQ.CARGOS.EDOSCTA,
XX DISP=OLD,DCB=BUFNO=2
54 XX DD DSN=HLQ.ABONOS.EDOSCTA,
XX DISP=OLD,DCB=BUFNO=2
55 XX DD DSN=HLQ.LEYENDA.PARACHQ,
XX DISP=OLD,DCB=BUFNO=2
56 XX DD DSN=HLQ.CMN615K.LEYENDAS,
XX DISP=OLD,DCB=BUFNO=2
57 XX DD DSN=HLQ.CMN615K.LEYENFH,
XX DISP=OLD,DCB=BUFNO=2
58 XX DD DSN=HLQ.LEYENDAS.JLS080,
XX DISP=OLD,DCB=BUFNO=2
59 XX DD DSN=HLQ.LEYEN.MULCASH,
XX DISP=OLD,DCB=BUFNO=5
60 XX DD DSN=HLQ.LINEA.OPERA.TIVA,
XX DISP=OLD,DCB=BUFNO=2
61 XX DD DSN=HLQ.LEYEN.MULCASH,
XX DISP=OLD,DCB=BUFNO=2
62 XX DD DSN=HLQ.LEYENDAS.ORDENES,
XX DISP=OLD,DCB=BUFNO=2
63 XX DD DSN=HLQ.EDOCTA,
XX DISP=OLD,DCB=BUFNO=5
64 XX DD DSN=HLQ.LEYENDAS.REMANEN,
XX DISP=OLD,DCB=BUFNO=2
65 XX DD DSN=HLQ.CHEQUES,
XX DISP=OLD,DCB=BUFNO=2
66 XX DD DSN=HLQ.CARCHEQ,
XX DISP=OLD,DCB=BUFNO=2
67 XX DD DSN=HLQ.DEBITO.EDOCTA,
XX DISP=OLD,DCB=BUFNO=2
68 XX DD DSN=HLQ.LEYEN.LIDE,
XX DISP=OLD,DCB=BUFNO=2
69 XXSORTOUT DD DSN=AMJOS.JDFDS.ACUM.TRANS.EDOSCTA,
XX UNIT=PRMDA,
XX DISP=(,CATLG,DELETE),
XX SPACE=(CYL,(2500,250),RLSE),
XX DCB=(MDLDCB.W150S.G00DX,BUFNO=30)
70 XXSYSIN DD *
SORT FIELDS=(1,25,A,32,23,A),FORMAT=BI

------------------------------------------------------------------------------------

ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE000I 1 - CONTROL STATEMENTS FOR 5740-SM1, DFSORT REL 14.0 - 23:32 ON FRI JUL 31, 2009 -
SORT FIELDS=(1,25,A,32,23,A),FORMAT=BI
ICE201I 0 RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE193I 0 ICEAM1 ENVIRONMENT IN EFFECT - ICEAM1 INSTALLATION MODULE SELECTED
ICE088I 2 PMXJDFDN.PMXJDFDN.SORT0005, INPUT LRECL = 150, BLKSIZE = 27900, TYPE = FB
ICE092I 0 MAIN STORAGE = (62914560,62914560,62914560)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (62523040,62523040)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=62914560,MAXLIM=1048576,MINLIM=450560,EQUALS=Y,LIST=Y,ERET=ABEND,MSGDDN=SYSOUT
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=N ,ABCODE=099
ICE130I 0 OPTIONS: RESALL=0,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,STIMER=Y,COBEXIT=COB1
ICE131I 0 OPTIONS: TMAXLIM=4194304,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DSA=0
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=N,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN
ICE750I 0 DC 11353179600 TC 0 CS DSVVV KSZ 52 VSZ 52
ICE752I 0 FSZ=75687864 RC IGN=0 E AVG=156 0 WSP=15335663 C DYN=0 0
ICE090I 0 OUTPUT LRECL = 150, BLKSIZE = 27900, TYPE = FB
ICE055I 0 INSERT 0, DELETE 0
ICE054I 0 RECORDS - IN: 75686433, OUT: 75686433
ICE134I 0 NUMBER OF BYTES SORTED: 11352964950
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 360000 , TRACKS USED: 210825
ICE180I 0 HIPERSPACE STORAGE USED = 953448K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE751I 0 C5C6C7C8E4C9E5E7CFCED3EDE8
ICE052I 0 END OF DFSORT

Escapa · Posted: Tue Aug 25, 2009 11:03 am

Singaram · New User Joined: 19 Aug 2009 Posts: 19 Location: India

Its taking around 1 hour elapsed..but Im trying to see is merge and then sort going to improve

Escapa · Posted: Tue Aug 25, 2009 12:13 pm

Singaram · New User Joined: 19 Aug 2009 Posts: 19 Location: India

CPU consumed is around 6 minutes

dick scherrer · Posted: Tue Aug 25, 2009 8:43 pm

Hello,

Are some or all of the concatenated input files already in sequence? If so, i suspect merging the files would be much faster than sorting. . .

Have the blksizes of the inputs/output been checked?

Singaram · New User Joined: 19 Aug 2009 Posts: 19 Location: India

Thanks dick scherrer for the reply

AMJOS.JDFDS.TRANS.EDOSCTA.DEPURADO this file alone is in sequence and it is having record count of say around 50million.

And regarding the blocksize can you please let me know what exactly you want to be checked.

dick scherrer · Posted: Tue Aug 25, 2009 9:11 pm

Hello,

As 50 million is 2/3s of the volume, sorting the others and then merging with these should be faster.

Talk with your storage management people to make sure the blksizes are efficient for the media being used.

Terry Heinze · Posted: Tue Aug 25, 2009 9:22 pm

No need to modify BUFNO -- DFSORT ignores yours and uses its own anyway. SORTWRK might be unnecessary too. Frank or Kolusu can either confirm or correct that assumption. Your 27900 BLKSIZE is as good as it's going to get. 1 hour seems reasonable considering you're sorting 75M records and using 6 CPU minutes. The DFSORT developers might have some tips for you.

Dave Betten · New User Joined: 24 Jan 2006 Posts: 26

First I'd like to point out that you are running DFSORT Release 14 which is no longer in support. So you may want to determine if you have a more current release like V1R5 or V1R10 available.

That said, you are sorting 75686433 records which add up to about 11GB and you used about 931MB of Hipespace. DFSORT does allocate its own buffers and used EXCP to read the input files.

Nothing jumps out at me from the sysout. If you have a system with more storage resources available, you could exploit more Hiperspace and reduce i/o to the work data sets. If you add //SORTDIAG DD DUMMY to the jcl, DFSORT will write out additional diagnostic messages. If you'd like to do that and send me the sysout, I'd be happy to see if I can find any additional tuning opportunity for this job. My email is betten@us.ibm.com