|
View previous topic :: View next topic
|
| Author |
Message |
MFsys
New User
Joined: 16 Feb 2018
Posts: 3
Location: India
|
|
|
|
Hello,
I am trying to solve a problem using DFSORT and wondering if it is even possible. I am able to achieve this using a REXX program, however, due to the volume of data, wondering if DFSORT could do it faster.
There are two columns containing TAPE volser for a dataset, which can extend on multiple tape volumes, but not in any particular order. So I need to find the head, tail and basically the entire chain. All I know is that head of any chain can only exist in first column with the next volume in the adjacent column.
The problem is similar to the one described in the question below.
ibmmainframes.com/viewtopic.php?t=58800&highlight=chain
Below is a small snippet of how the data might look like:
1ST VOL NEXT VOL
A00003 B00002
B00002 A00001
E00003 F00005
F00005 A00007
D00002 D00001
A00007 E00004
Expected Output
A00003 B00002 A00001
E00003 F00005 A00007 E00003
D00002 D00001 |
|
| Back to top |
|
 |
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1429
Location: Bamberg, Germany
|
|
|
|
| I don't think any SORT product is a good choice for your requirement. |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2264
Location: USA
|
|
|
|
With SORT it can be done only for a reasonably limited length of the chain.
In case you do not have 100,000-100,000,000 of volume names, using SORT in your case would give no benefits except disadvantages. |
|
| Back to top |
|
|
MFsys
New User
Joined: 16 Feb 2018
Posts: 3
Location: India
|
|
|
|
Thanks Joerg and sergeyken for the inputs. After having dig through the DFSORT manuals for possible solutions, I too felt the same (and learnt a lof of new things along the way), but obviously couldn't get to a solution (hence this post). Can you suggest any possible approaches you would take to solve it, if it MUST BE done using DFSORT.
And yes, the list of volumes in this case is quite high, roughly 400k on just one system. |
|
| Back to top |
|
|
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1429
Location: Bamberg, Germany
|
|
|
|
Somehow your input data seems to be strangely sorted (or non-sorted).
| Code: |
A00003 B00002
B00002 A00001
E00003 F00005
F00005 A00007
D00002 D00001
A00007 E00004 |
expected Output should be IMO:
| Code: |
A00003 B00002 A00001
E00003 F00005 A00007 E00004
D00002 D00001 |
right?
If this is a z/OS volume sequence, up to 255 tape volumes can be in a single chain. |
|
| Back to top |
|
|
MFsys
New User
Joined: 16 Feb 2018
Posts: 3
Location: India
|
|
|
|
Yes, the sample data I showed up was not sorted. I just made it up to bring forward the point that it is not necessary that A0001 may link to A0002 for example.
What you have presented above is correct.
I think you are right about the technical limitation of upto 255 volumes, I would have to look it up. But in this scenario, based on my analysis using REXX, the maximum depth of the tapes I found was about 10. I am not really sure if it would be a good idea to assume that as a limit (or maybe a little more, say 15), as long as the solution is dynamic and can be adapted for higher limit in future, I would want to learn about it. |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2264
Location: USA
|
|
|
|
The major trick is: how to use the same JOIN group for any number of join steps? Otherwise the total number of SORT control statements would exceed the size of input data 
| Code: |
//*====================================================================
// SET DEBUG='SYSOUT=*' intermediate results ON
//*SET DEBUG='DUMMY' intermediate results OFF
// SET LOG='SYSOUT=*' utility log ON
//*SET LOG='DUMMY' utility log OFF
// SET MSG='SYSOUT=*' utility messages ON
//*SET MSG='DUMMY' utility messages OFF
//*====================================================================
//CHAIN EXEC PGM=ICETOOL
//TOOLMSG DD &MSG
//DFMSG DD &LOG
//*
//SYMNAMES DD *
*----------------------------------------------------------------------
* Input file layout
*
FileName1,1,7,CH
FileName2,*,7,CH
*----------------------------------------------------------------------
* Joined file layout
*
JoinKey,1,7,CH
JoinPack,*,1993,CH
JoinLastName,*,7,CH
POSITION,JoinPack
JoinField,*,2000,CH
*-----------------------------------------------------------------------
//*
//SORTIN DD * test data; rearranged to simplify test verification
A00001 A00002
D00001 D00002
A00002 A00003
B00002 B00003
D00004 D00005
A00003 A00004
B00001 B00002
D00002 D00003
C00001 C00002
D00003 D00004
//*
//JOINTAB DD SPACE=(TRK,(100,100))
//*
//PREP1@ DD &DEBUG
//INIT1@ DD &DEBUG
//*
//JOIN1@ DD &DEBUG
//JOIN2@ DD &DEBUG
//JOIN3@ DD &DEBUG
//JOIN4@ DD &DEBUG
//JOIN5@ DD &DEBUG
//JOIN6@ DD &DEBUG
//JOIN7@ DD &DEBUG
//JOIN8@ DD &DEBUG
//JOIN9@ DD &DEBUG
//JOIN10@ DD &DEBUG
//*
//SORTOUT DD SYSOUT=*
//*
//*--------------------------------------------------------------------
//TOOLIN DD *
COPY FROM(SORTIN) TO(JOINTAB,PREP1@) USING(PREP)
COPY JKFROM TO(JOINTAB,INIT1@) USING(INIT)
COPY JKFROM TO(JOINTAB,JOIN1@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN2@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN3@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN4@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN5@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN6@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN7@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN8@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN9@) USING(JOIN)
COPY JKFROM TO(JOINTAB,JOIN10@) USING(JOIN)
* . . . . . . . etc. up to 255 times . . . . . . .
SORT FROM(JOINTAB) TO(SORTOUT) USING(SORT)
//*
//PREPCNTL DD *
OUTREC BUILD=(FileName2,FileName1,FileName2,
JoinLastName:7X)
//*
//INITCNTL DD *
JOINKEYS F1=JOINTAB,FIELDS=(JoinKey,A)
JOINKEYS F2=SORTIN,FIELDS=(FileName1,A)
JOIN UNPAIRED,F2,ONLY
REFORMAT FIELDS=(F2:FileName2,FileName1,FileName2)
OUTREC OVERLAY=(JoinLastName:7X)
//*
//JOINCNTL DD *
JOINKEYS F1=JOINTAB,FIELDS=(JoinKey,A)
JOINKEYS F2=SORTIN,FIELDS=(FileName1,A)
JOIN UNPAIRED,F1
REFORMAT FILL=C' ',
FIELDS=(F2:FileName2, =JoinKey
F1:JoinPack, =JoinPack
F2:FileName2) =JoinLastField
INREC BUILD=(JoinKey,
JoinField,SQZ=(SHIFT=LEFT,MID=C' '))
//*
//SORTCNTL DD *
SORT FIELDS=(JoinField,A)
OUTREC BUILD=(JoinField)
//*
//*==================================================================== |
| Code: |
********************************* TOP OF DATA ******************
A00001 A00002 A00003 A00004
B00001 B00002 B00003
C00001 C00002
D00001 D00002 D00003 D00004 D00005
******************************** BOTTOM OF DATA **************** |
|
|
| Back to top |
|
|
Rohit Umarjikar
Global Moderator
Joined: 21 Sep 2010
Posts: 3109
Location: NYC,USA
|
|
| Back to top |
|
|
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1429
Location: Bamberg, Germany
|
|
|
|
| @Rohit: One does not know if the source is DFRMM or CA1 (maybe something else). |
|
| Back to top |
|
|
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1429
Location: Bamberg, Germany
|
|
|
|
| The ICETOOL solution looks really nice, but wouldn't it be quite a bit of a SORT excess? Well, the other solution would have the opposite of IFTHEN excesses for what I have tried. I don't know if it's worth to be shared yet. |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2264
Location: USA
|
|
|
|
| Joerg.Findeisen wrote: |
| The ICETOOL solution looks really nice, but wouldn't it be quite a bit of a SORT excess? Well, the other solution would have the opposite of IFTHEN excesses for what I have tried. I don't know if it's worth to be shared yet. |
I tried some IFTHEN attempts, but the size of the required code is growing exponentially when the length of allowed chain increases.
If you have an acceptable solution, please share it with us. |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2264
Location: USA
|
|
|
|
With ICETOOL/SYNCTOOL some performance improvements can be done if needed. Like this one:
| Code: |
//JOINCNTL DD *
JOINKEYS F1=JOINTAB,FIELDS=(JoinKey,A)
JOINKEYS F2=SORTIN,FIELDS=(FileName1,A)
JOIN UNPAIRED,F1
REFORMAT FILL=C' ',
FIELDS=(F2:FileName2, =JoinKey
F1:JoinPack, =JoinPack
F2:FileName2) =JoinLastName
INREC IFTHEN=(WHEN=(JoinLastName,NE,C' '), <- often is empty
BUILD=(JoinKey,
JoinField,SQZ=(SHIFT=LEFT,MID=C' ')))
//* |
|
|
| Back to top |
|
|
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1429
Location: Bamberg, Germany
|
|
|
|
The bottleneck here is that the input must be in correct order for what I have attempted. I would like to read some comments if I am on the right path or not. It's just what came into my mind solving the requirement.
| Code: |
//WHATEVER EXEC PGM=ICEMAN
//F1 DD *
Z00000
A00003 B00002
B00002 A00001
E00003 F00005
F00005 A00007
A00007 E00004
X99999
D00002 D00001
/*
//F2 DD *
Z00000
A00003 B00002
B00002 A00001
E00003 F00005
F00005 A00007
A00007 E00004
X99999
D00002 D00001
/*
//SYSOUT DD SYSOUT=*
//SORTOUT DD SYSOUT=*
//SYSIN DD *
OPTION COPY
JOINKEYS F1=F1,FIELDS=(81,4,A)
JOINKEYS F2=F2,FIELDS=(81,4,A)
JOIN UNPAIRED,F1
REFORMAT FIELDS=(?,F1:1,6,8,6,F2:1,6)
INREC IFTHEN=(WHEN=GROUP,END=(8,6,CH,NE,14,6,CH),PUSH=(20:ID=4)),
IFTHEN=(WHEN=NONE,OVERLAY=(24:SEQNUM,4,ZD,RESTART=(20,4)))
OUTREC IFTHEN=(WHEN=GROUP,
BEGIN=(8,6,CH,NE,14,6,CH,AND,24,4,ZD,EQ,+1),
PUSH=(30:2,6,8,6),RECORDS=1),
IFTHEN=(WHEN=GROUP,
BEGIN=(8,6,CH,EQ,14,6,CH,AND,24,4,ZD,EQ,+1),
END=(8,6,CH,NE,14,6,CH),PUSH=(30:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+2),END=(8,6,CH,NE,14,6,CH),
PUSH=(36:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+3),END=(8,6,CH,NE,14,6,CH),
PUSH=(42:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+4),END=(8,6,CH,NE,14,6,CH),
PUSH=(48:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+5),END=(8,6,CH,NE,14,6,CH),
PUSH=(54:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+6),END=(8,6,CH,NE,14,6,CH),
PUSH=(60:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+7),END=(8,6,CH,NE,14,6,CH),
PUSH=(66:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+8),END=(8,6,CH,NE,14,6,CH),
PUSH=(72:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+9),END=(8,6,CH,NE,14,6,CH),
PUSH=(78:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+10),END=(8,6,CH,NE,14,6,CH),
PUSH=(84:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+11),END=(8,6,CH,NE,14,6,CH),
PUSH=(90:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+12),END=(8,6,CH,NE,14,6,CH),
PUSH=(96:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+13),END=(8,6,CH,NE,14,6,CH),
PUSH=(102:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+14),END=(8,6,CH,NE,14,6,CH),
PUSH=(108:2,6,8,6)),
IFTHEN=(WHEN=GROUP,BEGIN=(24,4,ZD,EQ,+15),END=(8,6,CH,NE,14,6,CH),
PUSH=(114:2,6,8,6))
OUTFIL FNAMES=(SORTOUT),
REMOVECC,NODETAIL,
SECTIONS=(20,4,
TRAILER3=(30,6,X,36,6,X,42,6,X,48,6,X,54,6,X,60,6,X,66,6,X,
72,6,X,78,6,X,84,6,X,90,6,X,96,6,X,102,6,X,108,6,X,
114,6))
END
/*
//JNF1CNTL DD *
INREC IFTHEN=(WHEN=INIT,OVERLAY=(81:SEQNUM,4,ZD,START=1))
END
/*
//JNF2CNTL DD *
INREC IFTHEN=(WHEN=INIT,OVERLAY=(81:SEQNUM,4,ZD,START=0))
END
/* |
Output:
| Code: |
****** ****************************
000001 Z00000
000002 A00003 B00002 A00001
000003 E00003 F00005 A00007 E00004
000004 X99999
000005 D00002 D00001
****** **************************** |
|
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2264
Location: USA
|
|
|
|
| Joerg.Findeisen wrote: |
| The bottleneck here is that the input must be in correct order for what I have attempted. I would like to read some comments if I am on the right path or not. |
In general case, when volume numbers are assigned randomly, I do not see a way to pre-arrange the lines in required order...
The rest of logic seems to work correctly. But the very basic operation - pre-arrangement of input lines - has been done manually, with no visible automation method? I believe so. |
|
| Back to top |
|
|
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1429
Location: Bamberg, Germany
|
|
|
|
| I have not yet looked deeper into pre-sorting stuff before our both approaches come into place. Maybe you have an idea that can be deployed? I am afraid to not have a good solution at the moment for that (possible) issue. |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2264
Location: USA
|
|
|
|
| Joerg.Findeisen wrote: |
| I have not yet looked deeper into pre-sorting stuff before our both approaches come into place. Maybe you have an idea that can be deployed? I am afraid to not have a good solution at the moment for that (possible) issue. |
Your approach works only when source lines are arranged in the order of their "chaining", which is Issue #1.
When this is done somehow, the Issue #2 is: combine multiple sequential lines into one single line, for each group of lines making a single chain.
Issue #2 can be resolved in one single pass, as you have done in your example.
Issue #1 cannot be resolved in one single pass, even theoretically. That's why in my approach multiple file scans have been used. |
|
| Back to top |
|
|
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1429
Location: Bamberg, Germany
|
|
|
|
Thank you for your reply. Let's assume the input is in the correct order.
In the wild, the situation is in general more complex with multivolume and multifile chains. Not really fun to work with from my experience. |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2264
Location: USA
|
|
|
|
| Joerg.Findeisen wrote: |
| The ICETOOL solution looks really nice, but wouldn't it be quite a bit of a SORT excess? |
Just because of curiosity, I tried to run the ICETOOL version with about 5000 of input lines (as many as I was not lazy to create).
The total job CPU time was 0.35 sec. (with 10 repeated JOIN attempt).
JOIN1 extends all needed chains from 2 to 3 names.
JOIN10 extends all needed chains from 11 to 12 names. |
|
| Back to top |
|
|
|
|
  |
All times are GMT + 6 Hours |
|
