Running in DFSORT V1R10 I was detect some SORT with SUM option, that increase a lot CPU consumption if is used Memory Objects.
Look a test, a file with a bit more of 6GB of size (with SUM option). First step use memory objects, second step Hiper (in both executions SORT was made 100% in memory, without use of DASD SORTWK* files)
JOBNAME: SCJLC0A , STEPNAME: PASO040M
BLOCKSET TECHNIQUE IN CONTROL
RECORD TYPE IS F - DATA STARTS IN POSITION 1
RA 0 WR 0 TR 0
C5-K90025 C6-K90025 C7-K54603 C8-K62201 E9-K60823 C9-BASE E5-K62201
BLOCKSET SORT TECHNIQUE SELECTED
VISIT www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AN
- CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R10 - 11:34 ON MON AP
SORT FIELDS=(27,26,BI,A,53,3,BI,A,74,10,BI,A,65,4,BI,A,
56,09,BI,A,69,5,BI,A,84,10,BI,A)
SUM FIELDS=(94,9,PD)
OPTION DYNALLOC=(,6),MOSIZE=MAX,HIPRMAX=0
RECORD TYPE IS F - DATA STARTS IN POSITION 1
RA 0 WR 0 TR 0
C5-K90025 C6-K90025 C7-K54603 C8-K62201 E9-K60823 C9-BASE E5-K62201
ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS
SCJLC0A .PASO040M. , INPUT LRECL = 132, BLKSIZE = 32736, TYPE =
MAIN STORAGE = (MAX,30941800,30924486)
MAIN STORAGE ABOVE 16MB = (30863960,30863960)
OPTIONS: OVFLO=RC16,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC4 ,VLSCMP=N,SZERO=Y,
OPTIONS: SIZE=30941800,MAXLIM=2097152,MINLIM=524288,EQUALS=N,LIST=Y,ER
OPTIONS: VIO=N,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=
OPTIONS: RESALL=12288,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT
OPTIONS: TMAXLIM=20971520,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW
OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=Y,TEXIT=N,LISTX=N,EFS=NONE ,EXITC
OPTIONS: HIPRMAX=0 ,DSPSIZE=0 ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMI
OPTIONS: NULLOUT=RC0
BSAM ACCESS METHOD USED FOR SORTOUT
BSAM ACCESS METHOD USED FOR SORTIN
DC 6600000000 TC 0 CS DSVUU KSZ 67 VSZ 67
FSZ=50000000 RC IGN=0 E AVG=132 0 WSP=8572266 C DYN=0 0
MOFSZ=6676,MOSZ=6676,MOSYS=101781(5),MOSTG=101781,MEML=17592186040320(
QP=418 QA=418 HI=6362 LI=5945 MI=6362 TZ=1709312 N1=1709056 N2=1709056
MOFR=0000,MOVR=VV
ESM=29146880,ESO=8744064,ESR=0,ESP=4096,ESS=16384,CES=29146880,HSZ=524
HWSP=4028320,HMAX=0,HES=0,ASV=26055936,EQ=I1,HN=0
OMAX=25015191,NMAX=26055936,ENQT=26055936,CMAX=0,HU=0,BUN=0,MD=N1,DU=0
CT=MAX , SB=4, L=0, D=0000, CCW=1MAM
W 01PP13 02PP13 03PP13 04PP13 05PP13 06PP13
O AZ00 I PP10
BA-K48846 BB-K69370 BC-BASE E8-K61438
CON=1,MUV=0,VOL=10,ENU=0,SBK=0,SRC=0,VEM=0
PWK=0 PSP=0 SWK=0 SSP=0 TWK=0 TSP=0 RWK=0 RSP=0 AWK=0 AWP=0
OUTPUT LRECL = 132, BLKSIZE = 32736, TYPE = FB
IN MAIN STORAGE SORT
INSERT 0, DELETE 49958210
RECORDS - IN: 50000000, OUT: 41790
NUMBER OF BYTES SORTED: 6600000000
RECORDS SORTED - PROCESSED: 50000000, EXPECTED: 50000000
TOTAL WORK DATA SET TRACKS ALLOCATED: 0 , TRACKS USED: 0
MEMORY OBJECT STORAGE USED = 6676M BYTES
HIPERSPACE STORAGE USED = 0K BYTES
DATA SPACE STORAGE USED = 0K BYTES
30915704 WMAIN, 8776 CMAIN, MAX CALLOC, N SCN, B BA, 0 AZ, 1 BZ, NN QC
132 RIN 32736 BLI 32736 BLO 132 RUN 56636 BUN 2817 CPU 00 CVC
255 XIN 127 WIN 50002094 GIN NDEY PFP BJD CMB00 CIX UPTH LMD VS RU
1 STR 0 MOR 0 IPB 200 OPB 0 CYL I MN
EQ=I1 DX=0 D2=0 D3=418 D4=0 AS=0 SA=0 SB=0 SC=0 HN=1
DAT 00 DSR 0304 BINS 50002094 BSZ 140 RCP 272 AJC 0 RLC 0 DUNIT 29
CW1 1 AVK 1 AVR 50000000
CW2 30 AVK 13 AVR 1666226
INIO1 255 AXIO1 255 W1 254 INIO3 200 AXIO3 200 BRUNS 1 VDN
0 MUNIT 0 SUNIT 0 OUNIT
0 SET 0 DEXTOT 0 BLK 50000000 CSZ 6 WE
SORTWK06 EXCP COUNT: 1
SORTWK05 EXCP COUNT: 1
SORTWK04 EXCP COUNT: 1
SORTWK03 EXCP COUNT: 1
SORTWK02 EXCP COUNT: 1
SORTWK01 EXCP COUNT: 1
END OF DFSORT
COMPLETE OUTPUT OF SECOND STEP (HIPER)
JOBNAME: SCJLC0A , STEPNAME: PASO040H
BLOCKSET TECHNIQUE IN CONTROL
RECORD TYPE IS F - DATA STARTS IN POSITION 1
RA 0 WR 0 TR 1
C5-K90025 C6-K90025 C7-K54603 C8-K62201 E9-K60823 C9-BASE E5-K62201
BLOCKSET SORT TECHNIQUE SELECTED
VISIT www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AN
- CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R10 - 11:40 ON MON AP
SORT FIELDS=(27,26,BI,A,53,3,BI,A,74,10,BI,A,65,4,BI,A,
56,09,BI,A,69,5,BI,A,84,10,BI,A)
SUM FIELDS=(94,9,PD)
OPTION DYNALLOC=(,6),MOSIZE=0,HIPRMAX=OPTIMAL
RECORD TYPE IS F - DATA STARTS IN POSITION 1
RA 0 WR 0 TR 1
C5-K90025 C6-K90025 C7-K54603 C8-K62201 E9-K60823 C9-BASE E5-K62201
ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS
SCJLC0A .PASO040H. , INPUT LRECL = 132, BLKSIZE = 32736, TYPE =
MAIN STORAGE = (MAX,30941800,30924486)
MAIN STORAGE ABOVE 16MB = (30863960,30863960)
OPTIONS: OVFLO=RC16,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC4 ,VLSCMP=N,SZERO=Y,
OPTIONS: SIZE=30941800,MAXLIM=2097152,MINLIM=524288,EQUALS=N,LIST=Y,ER
OPTIONS: VIO=N,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=
OPTIONS: RESALL=12288,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT
OPTIONS: TMAXLIM=20971520,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW
OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=Y,TEXIT=N,LISTX=N,EFS=NONE ,EXITC
OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=0 ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMI
OPTIONS: NULLOUT=RC0
BSAM ACCESS METHOD USED FOR SORTOUT
BSAM ACCESS METHOD USED FOR SORTIN
DC 6600000000 TC 0 CS DSVUU KSZ 67 VSZ 67
FSZ=50000000 RC IGN=0 E AVG=132 0 WSP=8572266 C DYN=1655 53216
MOFSZ=0,MOSZ=0,MOSYS=0(1),MOSTG=101776,MEML=17592186040320(3)
MOFR=0703,MOVR=VV
ESM=29146880,ESO=8744064,ESR=0,ESP=4096,ESS=16384,CES=29146880,HSZ=524
HWSP=4028320,HMAX=8388608,HES=8388608,ASV=26054801,EQ=I9,HN=1
OMAX=25003686,NMAX=26054801,ENQT=26054801,CMAX=524288,HU=99,BUN=53216,
QP=129 QA=129 HI=6362 LI=6234 MI=6362 TZ=524433 N1=524288 N2=524288 SZ
OMAX=25003686,NMAX=26054801,ENQT=25530368,CMAX=524288,HU=99,BUN=53216,
QP=128 QA=257 HI=6233 LI=6106 MI=6362 TZ=524288 N1=524288 N2=524288 SZ
OMAX=25003686,NMAX=26054801,ENQT=25006080,CMAX=524288,HU=99,BUN=53216,
QP=128 QA=385 HI=6105 LI=5978 MI=6362 TZ=524288 N1=524288 N2=524288 SZ
OMAX=25003686,NMAX=26054801,ENQT=24481792,CMAX=524288,HU=99,BUN=53216,
QP=128 QA=513 HI=5977 LI=5850 MI=6362 TZ=524288 N1=524288 N2=524288 SZ
OMAX=25003686,NMAX=26054801,ENQT=23957504,CMAX=45914,HU=99,BUN=53216,M
QP=12 QA=525 HI=5849 LI=5838 MI=6362 TZ=49152 N1=45914 N2=45914 SZ=13
CT=MAX , SB=4, L=0, D=0000, CCW=1MAM
W 03PP17 05PP17 02PP17 01PP17 04PP17 06PP13
O AZ00 I PP10
DE-K61785 D5-K62201 D3-BASE D7-BASE E8-K61438
CON=1,MUV=0,VOL=10,ENU=0,SBK=0,SRC=0,VEM=0
PWK=5 PSP=1655 SWK=0 SSP=0 TWK=0 TSP=0 RWK=0 RSP=0 AWK=5 AWP=1655
OUTPUT LRECL = 132, BLKSIZE = 32736, TYPE = FB
INSERT 0, DELETE 49958210
RECORDS - IN: 50000000, OUT: 41790
NUMBER OF BYTES SORTED: 6600000000
RECORDS SORTED - PROCESSED: 50000000, EXPECTED: 50000000
TOTAL WORK DATA SET TRACKS ALLOCATED: 1725 , TRACKS USED: 0
MEMORY OBJECT STORAGE USED = 0M BYTES
HIPERSPACE STORAGE USED = 766920K BYTES
DATA SPACE STORAGE USED = 0K BYTES
30915392 WMAIN, 9088 CMAIN, MAX CALLOC, N SCN, A BA, 0 AZ, 1 BZ, NN QC
132 RIN 32736 BLI 32736 BLO 132 RUN 53216 BUN 2817 CPU 00 CVC
203 XIN 14 WIN 161880 GIN NDEY PFP 3 BJD CMB00 CIX UPTS LMD VS RUX
164 STR 164 MOR 479 IPB 31 OPB 0 CYL O MN
EQ=I9 DX=0 D2=81 D3=46 D4=2 AS=0 SA=0 SB=0 SC=0 HN=1
DAT 00 DSR 0304 BINS 0 BSZ 0 RCP 0 AJC 0 RLC 0 DUNIT 0
407 MUNIT 42 SUNIT 31 OUNIT
968 SET 170535 DEXTOT 14513 BLK 50000000 CSZ 11 WE
SORTWK06 EXCP COUNT: 1
SORTWK05 EXCP COUNT: 2
SORTWK04 EXCP COUNT: 2
SORTWK03 EXCP COUNT: 2
SORTWK02 EXCP COUNT: 2
SORTWK01 EXCP COUNT: 2
HSR=1134,HSW=1104,HRE=14679,HWE=1745,HRP=190827,HWP=190827,HWM=191730,
END OF DFSORT
¿Need some PTF to solve it?
¿Is possible to avoid use of MO using SUM operations in a SORT? (V1R10), obviously maintaining other sort's no SUM, with possibility to use Memory Objects. In V1R5 problem doesn't exists, SUM operations are not selected to use with MO...
Many thanks in advance for your help and assistance, regards
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
I think it would help if you include the message references with your output. San Jose working-hours are several hours behind you, so this evening (earliest) or tomorrow for any response.
When re-posting, please use the Code tags, which preserve spacing and make things much easier to read. What version of z/OS are you running?
Below are my guesses based on behaviour you are seeing....
Josechu Lucena wrote:
Need some PTF to solve it?
Yes, you are probably missing some PTFs... UQ* ones. Wait for someone to provide detailed list of PTFs.
Josechu Lucena wrote:
Is possible to avoid use of MO using SUM operations in a SORT? (V1R10), obviously maintaining other sort's no SUM, with possibility to use Memory Objects. In V1R5 problem doesn't exists, SUM operations are not selected to use with MO...
Josechu
You have pretty much answered your question unless I am missing something. Hopefully once PTFs are installed, your issue may be resolved. In V1R5 you don't see this problem because PTFs that you are missing, are pre-installed with base(V1R5).
This has to do with the sort design being a bit different for memory object sorting and Hiperspace sorting. You have a large number of records with the same key value which causes the number of output records to be very small as compared to the number of input records.
When this situation occurs, memory object sorting can be a bit more cpu intensive. This was one of the reasons in PTF UK62201 (which you have), we altered our algorithms to prefer Hiperspace sorting more often. If instead of trying to force DFSORT to use one or the other with MOSIZE=0 or HIPRMAX=0, you left them at the shipped defaults of MOSIZE=MAX,HIPRMAX=OPTIMAL, DFSORT would probably choose Hiperspace sorting which as you've seen is more efficient in this case.
