Portal | Manuals | References | Downloads | Info | Programs | JCLs | Mainframe wiki | Quick Ref
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Profile Log in to check your private messages Log in
 
Performance Issue While SORTing

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL
View previous topic :: :: View next topic  
Author Message
ksk

Active User


Joined: 08 Jun 2006
Posts: 355
Location: New York

PostPosted: Wed Aug 06, 2008 2:47 pm    Post subject: Performance Issue While SORTing
Reply with quote

Hi,

One JOb in our shop is concatenating 5 files and sorting on the following creteria.

Code:

SORT FIELDS=(9,262,CH,A)
SUM FIELDS=NONE         


Actual record size is 271 and sort was done on the whole record except 1st 8 fileds. What I feel is 9 to 37 characters are quite enough to SORT the record as 38 to 271 has amount fields.

This single step is taking 3 hours clock time to execute. Could this happen due to length 262 characters mentioned in the SORT card or it is only due to removing duplicate records?
Back to top
View user's profile Send private message

gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Wed Aug 06, 2008 2:55 pm    Post subject:
Reply with quote

Hi,

can you provide the SYSOUT messages from the job and job log ?


Gerry
Back to top
View user's profile Send private message
ksk

Active User


Joined: 08 Jun 2006
Posts: 355
Location: New York

PostPosted: Wed Aug 06, 2008 3:27 pm    Post subject:
Reply with quote

Following is SYSOUT.

Code:

ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R5 - 00:40 ON WED AU
             SORT FIELDS=(9,262,CH,A)                                         
             SUM FIELDS=NONE                                                   
ICE201I E RECORD TYPE IS F - DATA STARTS IN POSITION 1                         
ICE751I 0 C5-K26318 C6-K90007 C7-K90000 C8-K23476 E9-K90007 C9-BASE   E5-K24705
ICE193I 0 ICEAM1 ENVIRONMENT IN EFFECT - ICEAM1 INSTALLATION MODULE SELECTED   
ICE088I 1 ZNDD803N.STEP10  .        , INPUT LRECL = 271, BLKSIZE = 27913, TYPE
ICE092I 0 MAIN STORAGE = (67108864,67108864,67108864)                         
ICE156I 0 MAIN STORAGE ABOVE 16MB = (66827704,66811320)                       
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC0 ,VLSCMP=N,SZERO=Y
ICE128I 0 OPTIONS: SIZE=67108864,MAXLIM=1048576,MINLIM=450560,EQUALS=Y,LIST=Y,E
ICE129I 0 OPTIONS: VIO=Y,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT
ICE130I 0 OPTIONS: RESALL=0,RESINV=0,SVC=109 ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,
ICE131I 0 OPTIONS: TMAXLIM=4194304,ARESALL=16384,ARESINV=0,OVERRGN=65536,CINV=Y
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXIT
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=0   ,ODMAXBF=0,SOLRF=N,VLLONG=N,VSAM
ICE235I 0 OPTIONS: NULLOUT=RC0                                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN                                   
ICE750I 0 DC 29338265693 TC 0 CS DSVUU KSZ 266 VSZ 266                         
ICE752I 0 FSZ=108259283 RC  IGN=0 E  AVG=276 0  WSP=38808415 C  DYN=705255 5321
ICE751I 1 DE-K24705 D5-K24705 D3-K24705 D7-K24705 E8-K24705                   
ICE090I 0 OUTPUT LRECL = 271, BLKSIZE = 27913, TYPE = FB                       
ICE055I 0 INSERT 0, DELETE 42507764                                           
ICE054I 0 RECORDS - IN: 108258986, OUT: 65751222                               
ICE134I 0 NUMBER OF BYTES SORTED: 29338185206                                 
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 705375 , TRACKS USED: 557595   
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES                               
ICE180I 0 HIPERSPACE STORAGE USED = 2097020K BYTES                             
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES                                   
ICE052I 0 END OF DFSORT                                                       
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Wed Aug 06, 2008 3:35 pm    Post subject:
Reply with quote

Hi,

you are sorting some large files, one way to speed this up and I'm sure there will other suggestions would be to have 5 jobs sorting individual files followed by a merge job.


Gerry
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Wed Aug 06, 2008 3:42 pm    Post subject:
Reply with quote

also see http://www-304.ibm.com/systems/support/storage/software/sort/mvs/tuning/


Gerry
Back to top
View user's profile Send private message
Dave Betten

New User


Joined: 24 Jan 2006
Posts: 26

PostPosted: Wed Aug 06, 2008 10:49 pm    Post subject: Reply to: Performance Issue While SORTing
Reply with quote

You are sorting 29GB of data and using 2GB of Hiperspace. So that means the remaining 27GB has to be written/read from DASD sortwork. So 3 hours is not entirely unreasonable but it's tough to say since I don't know what kind of I/O response times you are getting. This job is going to be heavily dependent on I/O response times for SORTIN, SORTWORK and SORTOUT.

Reducing the key field from 9,262 to 9,37 will help reduce CPU time but I don't think it will have a huge difference in elapsed.

I'd suggest you add //SORTDIAG DD DUMMY to the jcl for this sort and then send the sysout to dfsort@us.ibm.com. That's the email address for our DFSORT hotline and I can take a look and see if there's any tuning we can do for this sort. Be sure to send the entire sysout, not just the DFSORT messages.
Back to top
View user's profile Send private message
ksk

Active User


Joined: 08 Jun 2006
Posts: 355
Location: New York

PostPosted: Thu Aug 07, 2008 4:05 pm    Post subject:
Reply with quote

Hi Dave/Gerry,

Thanks for your immediate responses. I have increased work files and space and executed in test region with Production files, now it's taking 45 mins.

We will incorporate the same in production and if we face any problem I will let you know.

KSK
Back to top
View user's profile Send private message
ksk

Active User


Joined: 08 Jun 2006
Posts: 355
Location: New York

PostPosted: Thu Aug 07, 2008 4:09 pm    Post subject:
Reply with quote

Dave,

I forgot to ask one question. can you let me know what is the use of the following statement.

Code:

//SORTDIAG DD DUMMY
Back to top
View user's profile Send private message
Dave Betten

New User


Joined: 24 Jan 2006
Posts: 26

PostPosted: Thu Aug 07, 2008 5:35 pm    Post subject: Reply to: Performance Issue While SORTing
Reply with quote

The SORTDIAG causes additional diagnostic messages to be printed and these help us when performing indepth analysis. It does not do anything to improve performance so it has nothing to do with your decreased run time. How much Hiperspace did the sort use when you ran it in your test environment? It could be that there was less resource contention in the test environment and that's why it ran so much faster. Increasing work space reduces the risk of failure but shouldn't have that drastic an impact on elapsed time. Why don't you send me the entire sysout from that run and I can take a look.
Back to top
View user's profile Send private message
ksk

Active User


Joined: 08 Jun 2006
Posts: 355
Location: New York

PostPosted: Thu Aug 07, 2008 5:53 pm    Post subject:
Reply with quote

Here is the SYSOUT of changed JOB.

Code:

ICE201I E RECORD TYPE IS F - DATA STARTS IN POSITION 1                         
ICE751I 0 C5-K26318 C6-K90007 C7-K90000 C8-K23476 E9-K90007 C9-BASE   E5-K31980
ICE143I 0 BLOCKSET     SORT  TECHNIQUE SELECTED                                 
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AN
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R5 - 03:36 ON THU AUG
            SORT FIELDS=(9,21,CH,A)                                             
            SUM FIELDS=NONE                                                     
ICE201I E RECORD TYPE IS F - DATA STARTS IN POSITION 1                         
ICE751I 0 C5-K26318 C6-K90007 C7-K90000 C8-K23476 E9-K90007 C9-BASE   E5-K31980
ICE193I 0 ICEAM1 ENVIRONMENT IN EFFECT - ICEAM1 INSTALLATION MODULE SELECTED   
ICE088I 1 SSY361AB.STEP10  .        , INPUT LRECL = 271, BLKSIZE = 27913, TYPE =
ICE092I 0 MAIN STORAGE = (67108864,67108864,67108864)                           
ICE156I 0 MAIN STORAGE ABOVE 16MB = (66827704,66811320)                         
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC0 ,VLSCMP=N,SZERO=Y,
ICE128I 0 OPTIONS: SIZE=67108864,MAXLIM=1048576,MINLIM=450560,EQUALS=Y,LIST=Y,ER
ICE129I 0 OPTIONS: VIO=Y,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=
ICE130I 0 OPTIONS: RESALL=0,RESINV=0,SVC=109 ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,S
ICE131I 0 OPTIONS: TMAXLIM=4194304,ARESALL=16384,ARESINV=0,OVERRGN=65536,CINV=Y,
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXITC
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=0   ,ODMAXBF=0,SOLRF=N,VLLONG=N,VSAMI
ICE235I 0 OPTIONS: NULLOUT=RC0                                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT                                   
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN                                   
ICE750I 0 DC 29356353317 TC 0 CS DSVUU KSZ 25 VSZ 25                           
ICE752I 0 FSZ=108326027 RC  IGN=0 E  AVG=276 0  WSP=38832342 C  DYN=687315 53216
ICE915I 0 MOFSZ=0,MOSZ=0,MOSYS=2058(2),MOSTG=3072,MEML=2058(10)                 
ICE916I 0 MOFR=0304,MOVR=UU                                                     
ICE996I 0 ESM=786432,ESO=786432,ESR=437478,ESP=4096,ESS=16384,CES=4374784,HSZ=52
ICE997I 0 HWSP=18248281,HMAX=786432,HES=786432,ASV=786432,EQ=I2,HN=1           
ICE898I 0 OMAX=373004,NMAX=2269666,TMAX=2269666,CMAX=524288,HU=99,BUN=53216,MD=M
ICE880I 0 QP=128 QA=128 HI=192 LI=65 MI=192 TZ=524288 N1=524288 N2=524288 SZ=129
ICE889I 0 CT=MAX     , SB=8, L=0, D=0000                                       
ICE901I 0 W 0CPS13 01PS13 09PS13 0FPS13 04PS13 06PS13 07PS13 08PS13             
ICE901I 0 W 0EPS13 03PS13 05PS13 10PS13 0DPS13 0BPS13 0APS13 02PS13             
ICE902I 0 O PS10  I PS10                                                       
ICE897I 0 AVAL=0,BVAL=0,CVAL=0,ASV=0,HSZ=0,HM=M0,HN=3                           
ICE751I 1 DE-K24705 D5-K24705 D3-K24705 D7-K24705 E8-K24705                     
ICE900I 0 CON=5,MUV=0,VOL=132,ENU=0,SBK=5,SRC=223,VEM=0                         
ICE999I 0 PWK=15 PSP=687315 SWK=0 SSP=0 TWK=0 TSP=0 RWK=0 RSP=0 AWK=15 AWP=68731
ICE090I 0 OUTPUT LRECL = 271, BLKSIZE = 27913, TYPE = FB                       
ICE055I 0 INSERT 0, DELETE 42669466                                             
ICE054I 0 RECORDS - IN: 108325735, OUT: 65656269                               
ICE134I 0 NUMBER OF BYTES SORTED: 29356274185                                   
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 687375 , TRACKS USED: 534900   
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES                                 
ICE180I 0 HIPERSPACE STORAGE USED = 3144852K BYTES                             
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES                                   
ICE891I 1 67099672 WMAIN, 9192 CMAIN, MAX CALLOC, N SCN, A BA, 0 AZ, 2 BZ, NN QC
ICE892I 1 271 RIN  27913 BLI  27913 BLO  276 RUN  53216 BUN  2094 CPU  00 CVC   
ICE893I 1 200 XIN 14 WIN 210864 GIN NDEY PFP 5 B00 CM000 CIX UPTH LMD VS   RUX 
ICE894I 0 186 STR  186 MOR  1076 IPB  36 OPB  35660 CYL  O   MN                 
ICE881I 0 EQ=I2 DX=0 D2=0 D3=127 D4=1 AS=0 SA=0 SB=0 SC=0 HN=1                 
ICE885I 0 DAT 00 DSR 0304 BINS 0 BSZ 0 RCP 0 AJC 0 RLC 0 DUNIT 0               
ICE895I 0 443 MUNIT  42 SUNIT  31 OUNIT                                       
ICE896I 0 37247 SET  10666138 DEXTOT  558290 BLK   108326027 CSZ  69482 WE     
ICE804I 1 SORTWK10 EXCP COUNT: 5336                                           
ICE804I 1 SORTWK0F EXCP COUNT: 5349                                           
ICE804I 1 SORTWK0E EXCP COUNT: 5350                                           
ICE804I 1 SORTWK0D EXCP COUNT: 5331                                           
ICE804I 1 SORTWK0C EXCP COUNT: 5334                                           
ICE804I 1 SORTWK0B EXCP COUNT: 0                                               
ICE804I 1 SORTWK0A EXCP COUNT: 0                                               
ICE804I 1 SORTWK09 EXCP COUNT: 5340                                           
ICE804I 1 SORTWK08 EXCP COUNT: 5348                                           
ICE804I 1 SORTWK07 EXCP COUNT: 5346                                           
ICE804I 1 SORTWK06 EXCP COUNT: 5357                                           
ICE804I 1 SORTWK05 EXCP COUNT: 5349                                           
ICE804I 1 SORTWK04 EXCP COUNT: 5349                                           
ICE804I 1 SORTWK03 EXCP COUNT: 5351                                           
ICE804I 1 SORTWK02 EXCP COUNT: 0                                               
ICE804I 1 SORTWK01 EXCP COUNT: 5342                                             
ICE804I 1 SORTOUT  EXCP COUNT: 17805                                           
ICE804I 1 SORTIN   EXCP COUNT: 6660                                             
ICE899I 0 HSR=2729,HSW=2876,HRE=40189,HWE=4043,HRP=522457,HWP=522457,HWM=524242,
ICE052I 0 END OF DFSORT                                                         
******************************** BOTTOM OF DATA ********************************
Back to top
View user's profile Send private message
Dave Betten

New User


Joined: 24 Jan 2006
Posts: 26

PostPosted: Thu Aug 07, 2008 6:27 pm    Post subject: Reply to: Performance Issue While SORTing
Reply with quote

It would be better if you could send the entire sysout, not just the dfsort messages, as a text file to dfsort@us.ibm.com. Also if you can run the original sort with only the SORTDIAG change (not the sort control card and work space changes) and send that entire sysout as well? Then I can compare both and provide some feedback.
Back to top
View user's profile Send private message
ksk

Active User


Joined: 08 Jun 2006
Posts: 355
Location: New York

PostPosted: Fri Aug 08, 2008 11:09 am    Post subject:
Reply with quote

Dave,

Just I want to know an information from you on Hiperspace. Is hyperspace allocated by default by DFSORT or it would depend on other parameters like CLASS parameter, number of Work files, space allocated for work files and output files etc...

I would like to know because when I didn't use any Sort Work files it used 2GB hyperspace. When added Workfiles and more space, used 3GB. Today we changed CLASS parameter also apart from workfiles and space, it used Hyperspace 4GB.

With CLASS change, Work files and more space, above mentioned 2 steps are executed in 70 Mins in production.

I will send the entire Sysout of 2 cases in a day or 2.

KSK
Back to top
View user's profile Send private message
Dave Betten

New User


Joined: 24 Jan 2006
Posts: 26

PostPosted: Fri Aug 08, 2008 5:44 pm    Post subject: Reply to: Performance Issue While SORTing
Reply with quote

The amount of Hiperspace used depends on the amount of storage available to DFSORT at the time it executes. So it's quite possible there was just more available on the system at the time you ran the other sorts. Adding workspace does not effect how much Hiperspace we use.
Back to top
View user's profile Send private message
Guru Bob

New User


Joined: 31 Jan 2008
Posts: 21
Location: Malaysia

PostPosted: Fri Aug 15, 2008 3:16 pm    Post subject:
Reply with quote

My estimate is 20 SORTWK files at 2450 cylinders each. I assumed 3390 device going by the BLKSizes in the SORT output.

2GB for memory is good if you can get it all the time.

Even your output file requires 21,248 cylinders on a 3390.

Is the input on DISK or CART?
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts SIGNAL ON HALT issue packerm CLIST & REXX 1 Fri Oct 20, 2017 6:56 pm
No new posts Sorting on text - but in non-alphabet... Roy Ware SYNCSORT 5 Wed Aug 23, 2017 9:15 pm
No new posts CONTIG for performance? JPVRoff JCL & VSAM 8 Fri Jun 09, 2017 8:39 am
No new posts Sorting of hex values Saurabh_mi DFSORT/ICETOOL 11 Thu May 25, 2017 3:49 pm
No new posts Issue with NDM process to transmit ES... chetanambi All Other Mainframe Topics 6 Wed May 03, 2017 10:52 am

Facebook
Back to Top
 
Job Vacancies | Forum Rules | Bookmarks | Subscriptions | FAQ | Polls | Contact Us