IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Volume chain using DFSORT


IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
MFsys

New User


Joined: 16 Feb 2018
Posts: 3
Location: India

PostPosted: Tue Apr 11, 2023 11:38 pm
Reply with quote

Hello,

I am trying to solve a problem using DFSORT and wondering if it is even possible. I am able to achieve this using a REXX program, however, due to the volume of data, wondering if DFSORT could do it faster.

There are two columns containing TAPE volser for a dataset, which can extend on multiple tape volumes, but not in any particular order. So I need to find the head, tail and basically the entire chain. All I know is that head of any chain can only exist in first column with the next volume in the adjacent column.

The problem is similar to the one described in the question below.
ibmmainframes.com/viewtopic.php?t=58800&highlight=chain

Below is a small snippet of how the data might look like:

1ST VOL NEXT VOL
A00003 B00002
B00002 A00001
E00003 F00005
F00005 A00007
D00002 D00001
A00007 E00004
Expected Output

A00003 B00002 A00001
E00003 F00005 A00007 E00003
D00002 D00001
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1249
Location: Bamberg, Germany

PostPosted: Tue Apr 11, 2023 11:57 pm
Reply with quote

I don't think any SORT product is a good choice for your requirement.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2021
Location: USA

PostPosted: Wed Apr 12, 2023 12:13 am
Reply with quote

With SORT it can be done only for a reasonably limited length of the chain.

In case you do not have 100,000-100,000,000 of volume names, using SORT in your case would give no benefits except disadvantages.
Back to top
View user's profile Send private message
MFsys

New User


Joined: 16 Feb 2018
Posts: 3
Location: India

PostPosted: Wed Apr 12, 2023 2:36 am
Reply with quote

Thanks Joerg and sergeyken for the inputs. After having dig through the DFSORT manuals for possible solutions, I too felt the same (and learnt a lof of new things along the way), but obviously couldn't get to a solution (hence this post). Can you suggest any possible approaches you would take to solve it, if it MUST BE done using DFSORT.

And yes, the list of volumes in this case is quite high, roughly 400k on just one system.
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1249
Location: Bamberg, Germany

PostPosted: Wed Apr 12, 2023 1:20 pm
Reply with quote

Somehow your input data seems to be strangely sorted (or non-sorted).
Code:
A00003 B00002
B00002 A00001
E00003 F00005
F00005 A00007
D00002 D00001
A00007 E00004

expected Output should be IMO:
Code:
A00003 B00002 A00001
E00003 F00005 A00007 E00004
D00002 D00001

right?

If this is a z/OS volume sequence, up to 255 tape volumes can be in a single chain.
Back to top
View user's profile Send private message
MFsys

New User


Joined: 16 Feb 2018
Posts: 3
Location: India

PostPosted: Thu Apr 13, 2023 12:38 am
Reply with quote

Yes, the sample data I showed up was not sorted. I just made it up to bring forward the point that it is not necessary that A0001 may link to A0002 for example.

What you have presented above is correct.

I think you are right about the technical limitation of upto 255 volumes, I would have to look it up. But in this scenario, based on my analysis using REXX, the maximum depth of the tapes I found was about 10. I am not really sure if it would be a good idea to assume that as a limit (or maybe a little more, say 15), as long as the solution is dynamic and can be adapted for higher limit in future, I would want to learn about it.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2021
Location: USA

PostPosted: Thu Apr 13, 2023 6:59 am
Reply with quote

The major trick is: how to use the same JOIN group for any number of join steps? Otherwise the total number of SORT control statements would exceed the size of input data icon_lol.gif

Code:
//*====================================================================
// SET DEBUG='SYSOUT=*'     intermediate results ON                   
//*SET DEBUG='DUMMY'        intermediate results OFF                   
// SET LOG='SYSOUT=*'       utility log ON                             
//*SET LOG='DUMMY'          utility log OFF                           
// SET MSG='SYSOUT=*'       utility messages ON                       
//*SET MSG='DUMMY'          utility messages OFF                       
//*====================================================================
//CHAIN    EXEC PGM=ICETOOL                                           
//TOOLMSG  DD  &MSG                                                   
//DFMSG    DD  &LOG                                                   
//*                                                                   
//SYMNAMES DD  *                                                       
*----------------------------------------------------------------------
* Input file layout                                                   
*                                                                     
FileName1,1,7,CH                                                       
FileName2,*,7,CH                                                       
*----------------------------------------------------------------------
* Joined file layout                                                   
*                                                                     
JoinKey,1,7,CH                                                         
JoinPack,*,1993,CH                                                     
JoinLastName,*,7,CH                                                   
POSITION,JoinPack                                                     
JoinField,*,2000,CH                                                     
*-----------------------------------------------------------------------
//*                                                                     
//SORTIN   DD  *    test data; rearranged to simplify test verification                                                   
A00001 A00002                                                           
D00001 D00002                                                           
A00002 A00003                                                           
B00002 B00003                                                           
D00004 D00005                                                           
A00003 A00004                                                           
B00001 B00002                                                           
D00002 D00003                                                           
C00001 C00002                                                           
D00003 D00004                                                           
//*                                                                     
//JOINTAB  DD  SPACE=(TRK,(100,100))                                   
//*                                                                     
//PREP1@   DD  &DEBUG                                                   
//INIT1@   DD  &DEBUG                                                   
//*                                                                     
//JOIN1@   DD  &DEBUG                                                   
//JOIN2@   DD  &DEBUG                                                   
//JOIN3@   DD  &DEBUG                                                   
//JOIN4@   DD  &DEBUG                                                   
//JOIN5@   DD  &DEBUG                                                   
//JOIN6@   DD  &DEBUG                                                 
//JOIN7@   DD  &DEBUG                                                 
//JOIN8@   DD  &DEBUG                                                 
//JOIN9@   DD  &DEBUG                                                 
//JOIN10@  DD  &DEBUG                                                 
//*                                                                   
//SORTOUT  DD  SYSOUT=*                                               
//*                                                                   
//*--------------------------------------------------------------------
//TOOLIN   DD  *                                                       
 COPY FROM(SORTIN)   TO(JOINTAB,PREP1@)    USING(PREP)                 
 COPY JKFROM         TO(JOINTAB,INIT1@)    USING(INIT)                 
 COPY JKFROM         TO(JOINTAB,JOIN1@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN2@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN3@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN4@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN5@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN6@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN7@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN8@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN9@)    USING(JOIN)                 
 COPY JKFROM         TO(JOINTAB,JOIN10@)   USING(JOIN)   
* . . . . . . . etc. up to 255 times . . . . . . .             
 SORT FROM(JOINTAB)  TO(SORTOUT)           USING(SORT)                 
//*                                                                   
//PREPCNTL DD  *                                                       
 OUTREC BUILD=(FileName2,FileName1,FileName2,                     
               JoinLastName:7X)                                   
//*                                                               
//INITCNTL DD  *                                                   
 JOINKEYS F1=JOINTAB,FIELDS=(JoinKey,A)                           
 JOINKEYS F2=SORTIN,FIELDS=(FileName1,A)                           
 JOIN UNPAIRED,F2,ONLY                                             
 REFORMAT FIELDS=(F2:FileName2,FileName1,FileName2)               
 OUTREC OVERLAY=(JoinLastName:7X)                                 
//*                                                               
//JOINCNTL DD  *                                                   
 JOINKEYS F1=JOINTAB,FIELDS=(JoinKey,A)                           
 JOINKEYS F2=SORTIN,FIELDS=(FileName1,A)                           
 JOIN UNPAIRED,F1                                                 
 REFORMAT FILL=C' ',                                               
          FIELDS=(F2:FileName2,         =JoinKey                   
                  F1:JoinPack,          =JoinPack                 
                  F2:FileName2)         =JoinLastField             
 INREC BUILD=(JoinKey,                                             
              JoinField,SQZ=(SHIFT=LEFT,MID=C' '))                 
//*                                                               
//SORTCNTL DD  *                                                   
 SORT FIELDS=(JoinField,A)                                         
 OUTREC BUILD=(JoinField)                                         
//*                                                               
//*====================================================================


Code:
********************************* TOP OF DATA ******************
A00001 A00002 A00003 A00004                                     
B00001 B00002 B00003                                           
C00001 C00002                                                   
D00001 D00002 D00003 D00004 D00005                             
******************************** BOTTOM OF DATA ****************
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3051
Location: NYC,USA

PostPosted: Thu Apr 13, 2023 10:36 am
Reply with quote

Not sure if this helps to explore or try options to get formatted volume numbers .
www.ibm.com/docs/en/zos/2.2.0?topic=execs-tailoring-edgjrpt-sample-jcl
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1249
Location: Bamberg, Germany

PostPosted: Thu Apr 13, 2023 11:18 am
Reply with quote

@Rohit: One does not know if the source is DFRMM or CA1 (maybe something else).
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1249
Location: Bamberg, Germany

PostPosted: Thu Apr 13, 2023 11:25 am
Reply with quote

The ICETOOL solution looks really nice, but wouldn't it be quite a bit of a SORT excess? Well, the other solution would have the opposite of IFTHEN excesses for what I have tried. I don't know if it's worth to be shared yet.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2021
Location: USA

PostPosted: Thu Apr 13, 2023 4:49 pm
Reply with quote

Joerg.Findeisen wrote:
The ICETOOL solution looks really nice, but wouldn't it be quite a bit of a SORT excess? Well, the other solution would have the opposite of IFTHEN excesses for what I have tried. I don't know if it's worth to be shared yet.

I tried some IFTHEN attempts, but the size of the required code is growing exponentially when the length of allowed chain increases.

If you have an acceptable solution, please share it with us.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2021
Location: USA

PostPosted: Thu Apr 13, 2023 5:11 pm
Reply with quote

With ICETOOL/SYNCTOOL some performance improvements can be done if needed. Like this one:
Code:
//JOINCNTL DD  *                                                   
 JOINKEYS F1=JOINTAB,FIELDS=(JoinKey,A)                           
 JOINKEYS F2=SORTIN,FIELDS=(FileName1,A)                           
 JOIN UNPAIRED,F1                                                 
 REFORMAT FILL=C' ',                                               
          FIELDS=(F2:FileName2,         =JoinKey                   
                  F1:JoinPack,          =JoinPack                 
                  F2:FileName2)         =JoinLastName             
 INREC IFTHEN=(WHEN=(JoinLastName,NE,C' '),  <- often is empty
       BUILD=(JoinKey,                                             
              JoinField,SQZ=(SHIFT=LEFT,MID=C' ')))                 
//*   
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1249
Location: Bamberg, Germany

PostPosted: Thu Apr 13, 2023 5:22 pm
Reply with quote

The bottleneck here is that the input must be in correct order for what I have attempted. I would like to read some comments if I am on the right path or not. It's just what came into my mind solving the requirement.
Code:
//WHATEVER EXEC PGM=ICEMAN                                         
//F1       DD *                                                     
Z00000                                                             
A00003 B00002                                                       
B00002 A00001                                                       
E00003 F00005                                                       
F00005 A00007                                                       
A00007 E00004                                                       
X99999                                                             
D00002 D00001                                                       
/*                                                                 
//F2       DD *                                                     
Z00000                                                             
A00003 B00002                                                       
B00002 A00001                                                       
E00003 F00005                                                       
F00005 A00007                                                       
A00007 E00004                                                       
X99999                                                             
D00002 D00001                                                       
/*                                                                 
//SYSOUT   DD SYSOUT=*                                             
//SORTOUT  DD SYSOUT=*                                             
//SYSIN    DD *                                                     
  OPTION COPY                                                       
  JOINKEYS F1=F1,FIELDS=(81,4,A)                                   
  JOINKEYS F2=F2,FIELDS=(81,4,A)                                   
  JOIN UNPAIRED,F1                                                 
  REFORMAT FIELDS=(?,F1:1,6,8,6,F2:1,6)                             
  INREC IFTHEN=(WHEN=GROUP,END=(8,6,CH,NE,14,6,CH),PUSH=(20:ID=4)),
    IFTHEN=(WHEN=NONE,OVERLAY=(24:SEQNUM,4,ZD,RESTART=(20,4)))     
  OUTREC IFTHEN=(WHEN=GROUP,                                       
    BEGIN=(8,6,CH,NE,14,6,CH,AND,24,4,ZD,EQ,+1),                   
    PUSH=(30:2,6,8,6),RECORDS=1),                                   
    IFTHEN=(WHEN=GROUP,                                             
      BEGIN=(8,6,CH,EQ,14,6,CH,AND,24,4,ZD,EQ,+1),                 
      END=(8,6,CH,NE,14,6,CH),PUSH=(30:2,6,8,6)),                   
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+2),END=(8,6,CH,NE,14,6,CH),
      PUSH=(36:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+3),END=(8,6,CH,NE,14,6,CH),
      PUSH=(42:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+4),END=(8,6,CH,NE,14,6,CH),
      PUSH=(48:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+5),END=(8,6,CH,NE,14,6,CH),
      PUSH=(54:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+6),END=(8,6,CH,NE,14,6,CH),
      PUSH=(60:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+7),END=(8,6,CH,NE,14,6,CH),
      PUSH=(66:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+8),END=(8,6,CH,NE,14,6,CH),
      PUSH=(72:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+9),END=(8,6,CH,NE,14,6,CH),
      PUSH=(78:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+10),END=(8,6,CH,NE,14,6,CH),
      PUSH=(84:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+11),END=(8,6,CH,NE,14,6,CH),
      PUSH=(90:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+12),END=(8,6,CH,NE,14,6,CH),
      PUSH=(96:2,6,8,6)),                                             
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+13),END=(8,6,CH,NE,14,6,CH),
      PUSH=(102:2,6,8,6)),                                           
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+14),END=(8,6,CH,NE,14,6,CH),
      PUSH=(108:2,6,8,6)),                                           
    IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+15),END=(8,6,CH,NE,14,6,CH),
      PUSH=(114:2,6,8,6))                                             
  OUTFIL FNAMES=(SORTOUT),                                           
    REMOVECC,NODETAIL,                                               
    SECTIONS=(20,4,                                                   
      TRAILER3=(30,6,X,36,6,X,42,6,X,48,6,X,54,6,X,60,6,X,66,6,X,     
                72,6,X,78,6,X,84,6,X,90,6,X,96,6,X,102,6,X,108,6,X,   
                114,6))                                               
  END                                                                 
/*                                                                   
//JNF1CNTL DD *                                             
  INREC IFTHEN=(WHEN=INIT,OVERLAY=(81:SEQNUM,4,ZD,START=1)) 
  END                                                       
/*                                                           
//JNF2CNTL DD *                                             
  INREC IFTHEN=(WHEN=INIT,OVERLAY=(81:SEQNUM,4,ZD,START=0)) 
  END                                                       
/*

Output:
Code:
****** ****************************
000001 Z00000                     
000002 A00003 B00002 A00001       
000003 E00003 F00005 A00007 E00004
000004 X99999                     
000005 D00002 D00001               
****** ****************************
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2021
Location: USA

PostPosted: Thu Apr 13, 2023 6:02 pm
Reply with quote

Joerg.Findeisen wrote:
The bottleneck here is that the input must be in correct order for what I have attempted. I would like to read some comments if I am on the right path or not.

In general case, when volume numbers are assigned randomly, I do not see a way to pre-arrange the lines in required order...
The rest of logic seems to work correctly. But the very basic operation - pre-arrangement of input lines - has been done manually, with no visible automation method? I believe so.
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1249
Location: Bamberg, Germany

PostPosted: Thu Apr 13, 2023 6:28 pm
Reply with quote

I have not yet looked deeper into pre-sorting stuff before our both approaches come into place. Maybe you have an idea that can be deployed? I am afraid to not have a good solution at the moment for that (possible) issue.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2021
Location: USA

PostPosted: Thu Apr 13, 2023 10:32 pm
Reply with quote

Joerg.Findeisen wrote:
I have not yet looked deeper into pre-sorting stuff before our both approaches come into place. Maybe you have an idea that can be deployed? I am afraid to not have a good solution at the moment for that (possible) issue.

Your approach works only when source lines are arranged in the order of their "chaining", which is Issue #1.
When this is done somehow, the Issue #2 is: combine multiple sequential lines into one single line, for each group of lines making a single chain.

Issue #2 can be resolved in one single pass, as you have done in your example.

Issue #1 cannot be resolved in one single pass, even theoretically. That's why in my approach multiple file scans have been used.
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1249
Location: Bamberg, Germany

PostPosted: Thu Apr 13, 2023 11:40 pm
Reply with quote

Thank you for your reply. Let's assume the input is in the correct order.

In the wild, the situation is in general more complex with multivolume and multifile chains. Not really fun to work with from my experience.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2021
Location: USA

PostPosted: Fri Apr 14, 2023 9:00 pm
Reply with quote

Joerg.Findeisen wrote:
The ICETOOL solution looks really nice, but wouldn't it be quite a bit of a SORT excess?

Just because of curiosity, I tried to run the ICETOOL version with about 5000 of input lines (as many as I was not lazy to create).
The total job CPU time was 0.35 sec. (with 10 repeated JOIN attempt).
JOIN1 extends all needed chains from 2 to 3 names.
JOIN10 extends all needed chains from 11 to 12 names.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 


Similar Topics
Topic Forum Replies
No new posts Modifying Date Format Using DFSORT DFSORT/ICETOOL 9
No new posts Replace Multiple Field values to Othe... DFSORT/ICETOOL 12
No new posts Calling DFSORT from Cobol, using OUTF... DFSORT/ICETOOL 5
No new posts DFsort help with SUM() DFSORT/ICETOOL 12
No new posts DFSORT - VB file RDW getting overridden DFSORT/ICETOOL 3
Search our Forums:

Back to Top