Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

Sortwork and file space allocation mechanism
Goto page 1, 2  Next
 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL
View previous topic :: :: View next topic  
Author Message
liying

New User


Joined: 27 May 2010
Posts: 43
Location: da lian.china

PostPosted: Thu May 23, 2013 12:23 pm    Post subject: Sortwork and file space allocation mechanism
Reply with quote

hello:
I have faced the following problems and want to find the reason.

Problem 1 : the space I have assigned to a dataset was as follow.

Code:
//SORTOUT  DD  DSN=XXXX.DS.DATASET.D&TM,DISP=(,CATLG),
//             UNIT=(DISK2,6),
//             SPACE=(CYL,(300,100),RLSE),
//             MGMTCLAS=LQUARTR,
//             DCB=(RECFM=VBS,LRECL=256,BLKSIZE=27998)


and the dataset allocation info is as following.


Code:
Data Set Name . . . . : XXXX.DS.DATASET.D01                                   
                                                                             
General Data                           Current Allocation                     
 Management class . . : LQUARTR         Allocated cylinders : 4,396           
 Storage class  . . . : BASE            Allocated extents . : 4               
  Volume serial . . . : BDSE28 +                                             
  Device type . . . . : 3390                                                 
 Data class . . . . . :  無指定                                               
  Organization  . . . : PS             Current Utilization                   
  Record format . . . : VBS             Used cylinders  . . : 4,396           
  Record length . . . : 256             Used extents  . . . : 4               
  Block size  . . . . : 27998                                                 
  1st extent cylinders: 693                                                   
  Secondary cylinders : 100            Dates                                 
  Data set name type  :                 Creation date . . . : 2013/01/23     
  SMS Compressible. . : NO              Referenced date . . : 2013/05/23     
                                        Expiration date . . : ** なし **     


Code:
   Data Set Name . . . . : XXXX.DS.DATASET.D04                               
                                                                         
General Data                           Current Allocation               
 Management class . . : LQUARTR         Allocated cylinders : 4,638     
 Storage class  . . . : BASE            Allocated extents . : 2         
  Volume serial . . . : BDSEI5 +                                         
  Device type . . . . : 3390                                             
 Data class . . . . . :  無指定                                         
  Organization  . . . : PS             Current Utilization               
  Record format . . . : VBS             Used cylinders  . . : 4,638     
  Record length . . . : 256             Used extents  . . . : 2         
  Block size  . . . . : 27998                                           
  1st extent cylinders: 3331                                             
  Secondary cylinders : 100            Dates                             
  Data set name type  :                 Creation date . . . : 2013/04/19
  SMS Compressible. . : NO              Referenced date . . : 2013/05/23
                                        Expiration date . . : ** なし ** 


My question is : while the primary space has been assigned to be 300 cylinder, how it can range from 693 to 3331 cylinder. Meanwhile the second space has been assinged to be 100, how it can be 4,396 cylinder after 4 times extent?
Back to top
View user's profile Send private message

Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7224

PostPosted: Thu May 23, 2013 12:59 pm    Post subject: Reply to: sortwork and file space allocation mechenism
Reply with quote

The + near the volume-serial tells you it is on multiple volumes. What has this to do with "sortwork"?

What is RECFM=VBS for?

Can you show the full sysout from the step if this has been created by SORT?
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Thu May 23, 2013 1:04 pm    Post subject:
Reply with quote

Also bear in mind that the SMS allocation routines may override your JCL and make a completely different space allocation based on the DATACLAS definition.

As Bill has said, this is a multi volume dataset. Did you press enter and see just how many volumes have been used.
Back to top
View user's profile Send private message
liying

New User


Joined: 27 May 2010
Posts: 43
Location: da lian.china

PostPosted: Thu May 23, 2013 2:11 pm    Post subject:
Reply with quote

expat wrote:
Also bear in mind that the SMS allocation routines may override your JCL and make a completely different space allocation based on the DATACLAS definition.

As Bill has said, this is a multi volume dataset. Did you press enter and see just how many volumes have been used.


as it had been defined [UNIT=(DISK2,6)],6 volume have been used

Code:
                              Data Set Information                       
 C Esssssssssss Volume Information sssssssssssN                         
   e                                          e                         
 D e Command ===>                             e                         
   e                                          e                         
 G e All allocated volumes:                   e Allocation               
   e                             続く      +  e ed cylinders : 4,396     
   e  Number of volumes allocated: 6          e ed extents . : 4         
   e                                          e                         
   e  BDSE28  BDSN68  *       *       *       e                         
   e  *                                       e                         
   e                                          e Utilization             
   e  F1=Help      F2=Split     F3=Exit       e linders  . . : 4,396     
   e  F7=Backward  F8=Forward   F9=Swap       e tents  . . . : 4     
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Thu May 23, 2013 2:22 pm    Post subject:
Reply with quote

No, only two of the possible 6 volumes have been used.
Back to top
View user's profile Send private message
liying

New User


Joined: 27 May 2010
Posts: 43
Location: da lian.china

PostPosted: Thu May 23, 2013 2:36 pm    Post subject:
Reply with quote

expat wrote:
No, only two of the possible 6 volumes have been used.


but the information shows that there were 6 volums have been allocated.
whatever, how can the 1st extent and 2nd extend far more large than the number I have assigned
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7224

PostPosted: Thu May 23, 2013 2:40 pm    Post subject: Reply to: sortwork and file space allocation mechenism
Reply with quote

If you want answers from us, you have to give answers and information from all our questions.

If you want to know why your extents have been "consolidated", ask your storage-management people, they know what you have, with what options, and knowing that you will have an explanation for that.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Thu May 23, 2013 2:45 pm    Post subject:
Reply with quote

The display shows that only 2 of the possible 6 volumes have been used. The asterisk indicates that there are 4 more volumes available for the dataset to expand onto if required.

Did you notice the difference between the creation date and the last referenced date.

Possibilities are -
HSM has processed these datasets during the HSM housekeeping cycle and has used the extent reduction process to consolidate the dataset into the minimum number of extents that it could.

Has the dataset been migrated and then recalled - HSM will reallocate the dataset in as few extents as possible.
Back to top
View user's profile Send private message
liying

New User


Joined: 27 May 2010
Posts: 43
Location: da lian.china

PostPosted: Thu May 23, 2013 2:46 pm    Post subject: Re: Reply to: sortwork and file space allocation mechenism
Reply with quote

Bill Woodger wrote:
If you want answers from us, you have to give answers and information from all our questions.

If you want to know why your extents have been "consolidated", ask your storage-management people, they know what you have, with what options, and knowing that you will have an explanation for that.


Sorry, I put the all system output in the following. VBS stand for file's format is Variable Blocked Spanned.

Code:
 VPW//PRT010   DD SYSOUT=*,DCB=(RECFM=VBA)
1                    OPTION SORTINS=2
                     RECORD TYPE=F,LENGTH=(250)
                     SORT FIELDS=(1,6,CH,A)
0         * PHASE 1           INPUT      100,569,450
                              DELETE               0
                              USED       100,569,450
          * PHASE 3           SUMMARY              0
                              OUTPUT     100,569,450
 VPW//PRT011   DD SYSOUT=*,DCB=(RECFM=VBA)
1ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
 ICE118I 0 UNKNOWN FILE SIZE - FILSZ=EN MAY IMPROVE RESOURCE USAGE AND PERFORMANCE
 ICE751I 0 C5-K76982 C6-K90026 C7-K82419 C8-K67572 E4-K58148 C9-BASE   E5-K80744 E7-K79990
 ICE143I 0 BLOCKSET     SORT  TECHNIQUE SELECTED
 ICE250I 0 VISIT クホホニ://ムムム.ケイテ.ウナテ/ヘホナネアキオ/エカヘナネホ FOR DFSORT PAPERS, EXAMPLES AND MORE
 ICE000I 0 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 03:03 ON THU MAY 23, 2013 -
0           SORT FIELDS=(1,6,CH,A)
            RECORD TYPE=F,LENGTH=(250)
 ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
 ICE118I 0 UNKNOWN FILE SIZE - FILSZ=EN MAY IMPROVE RESOURCE USAGE AND PERFORMANCE
 ICE751I 0 C5-K76982 C6-K90026 C7-K82419 C8-K67572 E4-K58148 C9-BASE   E5-K80744 E7-K79990
 ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
 ICE252I 1 PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS
 ICE089I 1 OS03QM10.SORT20  .A       , INPUT LRECL = 250, TYPE = F
 ICE093I 0 MAIN STORAGE = (MAX,6291456,6291456)
 ICE156I 0 MAIN STORAGE ABOVE 16MB = (6234096,6234096)
 ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=32
 ICE128I 0 OPTIONS: SIZE=6291456,MAXLIM=2097152,MINLIM=450560,EQUALS=Y,LIST=Y,ERET=ABEND,MSGDDN=SYSOUT
 ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=N             ,ABCODE=MSG
 ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
 ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=0
 ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=N,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
 ICE133I 0 OPTIONS: HIPRMAX=8      ,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
 ICE235I 0 OPTIONS: NULLOUT=RC0
 ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y
 ICE750I 0 DC 0 TC 0 CS DSVVV KSZ 10 VSZ 10
 ICE752I 0 FSZ=0 RE  IGN=0 C  AVG=256 0  WSP=0 E  DYN=0 0
 ICE751I 1 DE-K83743 D5-K58148 D3-K83080 D7-K58148 E8-K79990
 ICE091I 0 OUTPUT LRECL = 250, TYPE = F
 ICE055I 0 INSERT 100569450, DELETE 100569450
 ICE054I 0 RECORDS - IN: 0, OUT: 0
 ICE134I 0 NUMBER OF BYTES SORTED: 25142362500
 ICE253I 0 RECORDS SORTED - PROCESSED: 100569450, EXPECTED: 0
 ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 505710 , TRACKS USED: 500055
 ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
 ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 1937M BYTES
 ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES
 ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
 ICE052I 0 END OF DFSORT
Back to top
View user's profile Send private message
Pete Wilson

Active User


Joined: 31 Dec 2009
Posts: 437
Location: London

PostPosted: Thu May 23, 2013 2:54 pm    Post subject:
Reply with quote

This is quite possible for a number of reasons, and I'd suggest you don't get to worried about it. As long as the file is big enough and you're not getting space failures what is the issue?

If you allocate 300,100 cyls space and unit count of 6, then a standard dataset can have the primary allocation plus 15 secondary on each of the volumes, or if it is Extended Format (as determined by Dataclas) it can have up to 123 extents per volume. If it has a guaranteed-Space storclas will also allocate primary first on each of the volumes before taking secondary extents.

When you view it in 3.4 the '1st extent cylinders' field shows the size of the first extent of the primary allocation only. This can vary because of things like:
a) extent conslidation where SMS will join extents together if there are tracks free next to one extent when it takes another extent.
b) how fragmented the volume is at allocation time
c) if you run some form of defrag of the volume which may combine extents together
d) if it's been migrated by DFHSM it will try and allocate the full size of the dataset as primary on the recall.
c) Space Constraint Relief (a Dataclas field) can cut allocations down to make it fit into available gaps so the value for primary in 3.4 could look different every time it's allocated.
Back to top
View user's profile Send private message
liying

New User


Joined: 27 May 2010
Posts: 43
Location: da lian.china

PostPosted: Thu May 23, 2013 3:13 pm    Post subject:
Reply with quote

Pete Wilson wrote:
This is quite possible for a number of reasons, and I'd suggest you don't get to worried about it. As long as the file is big enough and you're not getting space failures what is the issue?

If you allocate 300,100 cyls space and unit count of 6, then a standard dataset can have the primary allocation plus 15 secondary on each of the volumes, or if it is Extended Format (as determined by Dataclas) it can have up to 123 extents per volume. If it has a guaranteed-Space storclas will also allocate primary first on each of the volumes before taking secondary extents.

When you view it in 3.4 the '1st extent cylinders' field shows the size of the first extent of the primary allocation only. This can vary because of things like:
a) extent conslidation where SMS will join extents together if there are tracks free next to one extent when it takes another extent.
b) how fragmented the volume is at allocation time
c) if you run some form of defrag of the volume which may combine extents together
d) if it's been migrated by DFHSM it will try and allocate the full size of the dataset as primary on the recall.
c) Space Constraint Relief (a Dataclas field) can cut allocations down to make it fit into available gaps so the value for primary in 3.4 could look different every time it's allocated.


perfect , thank you so much.
Would kindly please tell me where you have got this information? Is there any reference or guides?
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7224

PostPosted: Thu May 23, 2013 5:54 pm    Post subject: Reply to: sortwork and file space allocation mechenism
Reply with quote

I know what VBS is, I was wondering how you were managing to use it :-)

From the sysout, both your input and output are being treated a F, not even FB. I'm not sure what you are getting on your output, I've never tried that :-)

You may have some performance issues.

What is the DCB of the input?

Again, knowing what VBS is, why are you using it for what you have told SORT is fixed-length records?

Usually Spanned records are larger than the blocksize. If they are not, there is not much point in using Spanned over Variable, that I can think of, anyway. Even if variable, why have them all as fixed-length - that is an extra four bytes per record, not isiginificant when you have 100,000,000+ records :-)
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 7905
Location: Bellevue, IA

PostPosted: Thu May 23, 2013 6:03 pm    Post subject:
Reply with quote

Quote:
Usually Spanned records are larger than the blocksize. If they are not, there is not much point in using Spanned over Variable, that I can think of, anyway.
Bill: I had a case (many, many years ago) where spanned helped for an application. I had a fixed set of employee data, followed by a variable number of job histories. The fixed data was about 1000 bytes and the variable portion pushed the LRECL up over 22,000. At the time, the operating system wrote variable-length blocks when the remaining space in the block was not large enough to hold a complete record, so effectively I was leaving thousands and tens of thousands of bytes unused in each block. By using spanned, the system would leave at most 4 bytes unused in the block and vastly decreased the number of blocks used.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7224

PostPosted: Thu May 23, 2013 6:11 pm    Post subject: Reply to: sortwork and file space allocation mechenism
Reply with quote

Thanks Robert.

For readers, these days APPLY WRITE-ONLY deals with that problem from COBOL, the block is only written when the current record written is too long for the remaining space in the block (not the maximum-length record (LRECL)).

In this topic I can't 1) see what VBS gets (LRECL of 256), 2) see why to treat them in SORT as F, 3) see why override the output dataset as VBS again (again 256 LRECL, whereas F 250 in the SORT) - and I don't even know what that will do :-)
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 7905
Location: Bellevue, IA

PostPosted: Thu May 23, 2013 6:19 pm    Post subject:
Reply with quote

The data posted so far is definitely inconsistent, and I'm surprised the application can use the output of the SORT anyway. VBS with LRECL of 256 implies a record length of 252, yet SORT is outputting 250 bytes fixed length records. And yet the only concern is about the allocation of extents?
Back to top
View user's profile Send private message
Pete Wilson

Active User


Joined: 31 Dec 2009
Posts: 437
Location: London

PostPosted: Thu May 23, 2013 6:26 pm    Post subject:
Reply with quote

Quote:
Would kindly please tell me where you have got this information? Is there any reference or guides?


Look at IBM manuals:

- Using Datasets
- DFSMS Storage Administration Reference (and the Guide)


I figured out what the Primary space value was in 3.4 by looking at a phsyical volume map of the volume a file was allocated on (IEHLIST or Fileaid 3.7 for example), because I was mystified how it was showing such numbers.
Back to top
View user's profile Send private message
liying

New User


Joined: 27 May 2010
Posts: 43
Location: da lian.china

PostPosted: Thu May 23, 2013 6:39 pm    Post subject: Re: Reply to: sortwork and file space allocation mechenism
Reply with quote

Bill Woodger wrote:
I know what VBS is, I was wondering how you were managing to use it :-)

From the sysout, both your input and output are being treated a F, not even FB. I'm not sure what you are getting on your output, I've never tried that :-)

You may have some performance issues.

What is the DCB of the input?

Again, knowing what VBS is, why are you using it for what you have told SORT is fixed-length records?

Usually Spanned records are larger than the blocksize. If they are not, there is not much point in using Spanned over Variable, that I can think of, anyway. Even if variable, why have them all as fixed-length - that is an extra four bytes per record, not isiginificant when you have 100,000,000+ records :-)


Thank you for your patient explanation. The reason to use VBS is that, the file in FB format is too large, and there are many blanks in the content of the dataset, therefore use VBS dataset can save huge space for me.(In face, in my application, VBS dataset have been offen used to storage data which can save a lot of space, you can have a try that use DCPVV1 to transfrom a dataset in FB format which have many blanks to VBS format, it may only use less than 20% storage of the original FB dataset, in my case, the record lenght was 250, and the recoreds number are 100,569,450, and in FB dataset it needs almost 30,000 cylinders whereas VBS file is only 4700 cylinders)

The input file was also VBS file.
Back to top
View user's profile Send private message
liying

New User


Joined: 27 May 2010
Posts: 43
Location: da lian.china

PostPosted: Thu May 23, 2013 6:45 pm    Post subject: Re: Reply to: sortwork and file space allocation mechenism
Reply with quote

Bill Woodger wrote:
Thanks Robert.

For readers, these days APPLY WRITE-ONLY deals with that problem from COBOL, the block is only written when the current record written is too long for the remaining space in the block (not the maximum-length record (LRECL)).

In this topic I can't 1) see what VBS gets (LRECL of 256), 2) see why to treat them in SORT as F, 3) see why override the output dataset as VBS again (again 256 LRECL, whereas F 250 in the SORT) - and I don't even know what that will do :-)


1) input file is VBS 256,
2) because there are keys to be sorted has been specified, I am not sure the result is right to treat them in SORT as V
3)to overrige the output from F to VBS is to save the storage.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7224

PostPosted: Thu May 23, 2013 6:55 pm    Post subject: Reply to: sortwork and file space allocation mechenism
Reply with quote

But why not just make it VB? Why the "spanned"?

You are getting this:

Quote:
ICE118I 0 UNKNOWN FILE SIZE - FILSZ=EN MAY IMPROVE RESOURCE USAGE AND PERFORMANCE


On top of that, you used 33,337 cylinders of sortwork space. If by using variable-length records you get that down to 4,700 cylinders, that alone is going to improve things greatly for this step.

So, why VBS, not VB, why F not VB, why VBS in-and-out but treat it as FB in the SORT?
Back to top
View user's profile Send private message
liying

New User


Joined: 27 May 2010
Posts: 43
Location: da lian.china

PostPosted: Thu May 23, 2013 7:06 pm    Post subject: Re: Reply to: sortwork and file space allocation mechenism
Reply with quote

Bill Woodger wrote:
But why not just make it VB? Why the "spanned"?

You are getting this:

Quote:
ICE118I 0 UNKNOWN FILE SIZE - FILSZ=EN MAY IMPROVE RESOURCE USAGE AND PERFORMANCE


On top of that, you used 33,337 cylinders of sortwork space. If by using variable-length records you get that down to 4,700 cylinders, that alone is going to improve things greatly for this step.

So, why VBS, not VB, why F not VB, why VBS in-and-out but treat it as FB in the SORT?


Thank you very much, I will do some test based on your suggestion. :-)
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL All times are GMT + 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Add PD field from 2nd file to PD in 1st Sushant Garje DFSORT/ICETOOL 6 Thu Dec 01, 2016 4:32 pm
No new posts File Aid to File Manager conversion murali3955 IBM Tools 4 Thu Nov 24, 2016 3:41 pm
No new posts How to convert the VBM file to VB or... Sulabh Agrawal JCL & VSAM 4 Fri Nov 18, 2016 1:04 pm
No new posts CICS Roll back partially - Need to re... dwijadas CICS 4 Wed Nov 16, 2016 4:30 pm
No new posts Problem in writing Output file vickey_dw COBOL Programming 5 Mon Nov 14, 2016 11:14 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us