Joined: 07 Feb 2009 Posts: 1306 Location: Vilnius, Lithuania
The problem:
Before changes to one particular program are made live, I run regression tests, and in the fast I've been able to fall back on SuperC to mask changes, on one occasion I went from '.' to ':' as time separator characters, on another a "+---+---..." separators were changed into "+===+===...", and the "nchgt" option of SuperC took care of those differences.
However, this time I've finally sorted out a very long standing fudge, and that means the input file format is about to change, in a subtle way, but not subtle enough to be transparent to older versions of the program, and so I'd like to automate the conversion of format "n" input data to format "n+1". The key change is that I'm going to get rid of pseudo-intervals, that are currently used to record idle-time.
In other words, if a group of records currently contains the following records, here limited to some key data:
I need (Not long ago I found that it has unintended side-effects) to get rid of the "!^" pseudo-interval, and change the actual time in the first record of the group accordingly, i.e:
There are (currently) 437 of these pseudo-intervals present in the input data, and it would be nice it there was a way to let DFSORT do this conversion, so if anyone has any ideas, I'd love to see them.
For what it's worth, I'm not (too) hung up on using DFSORT, if it gets too complicated (a set of "!^&" intervals must stay, and about a dozen others, but I'm sure that that can be taken care of by a few extra IFTHEN=(WHEN... statements), I can just write a small pre-processor for the data to (for now, until version n+3 of the program) convert it into the new format.
Joined: 17 Oct 2006 Posts: 2481 Location: @my desk
prino,
Is there a way to identify the first record of each 'group'? Can there be multiple "!^" pseudo-interval records in one 'group'? If yes which time should go to the first record?
Joined: 07 Feb 2009 Posts: 1306 Location: Vilnius, Lithuania
Arun,
No, there can only be one "!^" record in a group, it is always the last in the group, it's always in columns 48-49, and, somewhat redundant, column 66 will also contain a "!".
A valid group starts with "nnn," in cols 1-4, where nnn is a right aligned number without leading zero's between 1 and (theoretically) 999, and has a blank in column 66.
No, there can only be one "!^" record in a group, it is always the last in the group, it's always in columns 48-49, and, somewhat redundant, column 66 will also contain a "!".
A valid group starts with "nnn," in cols 1-4, where nnn is a right aligned number without leading zero's between 1 and (theoretically) 999, and has a blank in column 66.
Joined: 07 Feb 2009 Posts: 1306 Location: Vilnius, Lithuania
sergeyken wrote:
prino wrote:
Arun,
No, there can only be one "!^" record in a group, it is always the last in the group, it's always in columns 48-49, and, somewhat redundant, column 66 will also contain a "!".
A valid group starts with "nnn," in cols 1-4, where nnn is a right aligned number without leading zero's between 1 and (theoretically) 999, and has a blank in column 66.
Why not just to OMIT COND=(x,y,CH,EQ,C'!^') - ?
Because the start-time from the "!^" record needs to be carried back to the end-time of the last parent record. See the example!
Joined: 17 Oct 2006 Posts: 2481 Location: @my desk
prino wrote:
A valid group starts with "nnn," in cols 1-4, where nnn is a right aligned number without leading zero's between 1 and (theoretically) 999, and has a blank in column 66.
Hi prino,
Do you mind showing an example that consists of 'valid' and 'invalid groups'?
If we are able to identify 'valid groups', then probably we can do it by joining the file to itself and then attach the value from the last record onto the first record.
By removing the "!^" record from the above, as done below, without changing the actual end-time on the header (i.e.: "column 66 is blank") record of the "69, 18" group, that group becomes invalid, because the 1.02 time no longer matches 10.55 - 8.27 - (8.47 - 8.35) - (9.41 - 9.35)
The correction of the end-time on the "header" of the "69, 18" group to 9.47 below might seem to generate an inconsistency, as 9.47 + 0.15 (from the "69, 19" group) no longer matches the 11.10 starting time of that group. However, the code has now been changed to cater for such, dumping the unaccounted for time into the same (idle) bin as "!^" time, and not, as what's used to happen, the time-zone bin.
The change (of removing all "!^" records, and I'm sadly still stuck with a few, sigh...) is required, because some "!^" intervals take a collection of consecutive groups over 24 hours, when in reality the period is under 24 hours, and should be reported as such.
Joined: 17 Oct 2006 Posts: 2481 Location: @my desk
prino,
Assuming the input to be of FB/LRECL=100, the below DFSORT step should work for your requirement. Also assumed the number field to be at position 68 and length-6.
Code:
//STEP001 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTJNF1 DD DISP=SHR,DSN= Input data set (FB/LRECL=100)
//SORTJNF2 DD DISP=SHR,DSN= Same Input data set (FB/LRECL=100)
//SORTOUT DD DISP=SHR,DSN= Output data set (FB/LRECL=100)
//SYSIN DD *
JOINKEYS FILE=F1,FIELDS=(101,9,A),SORTED,NOSEQCK
JOINKEYS FILE=F2,FIELDS=(101,9,A),SORTED,NOSEQCK
JOIN UNPAIRED,F1
REFORMAT FIELDS=(F1:1,100,F2:68,6,?)
OMIT COND=(48,2,CH,EQ,C'!^')
INREC IFOUTLEN=100,
IFTHEN=(WHEN=(66,1,CH,EQ,C' ',AND,107,1,CH,EQ,C'B'),
OVERLAY=(75:101,6))
SORT FIELDS=COPY
//JNF1CNTL DD *
INREC IFTHEN=(WHEN=GROUP,BEGIN=(66,1,CH,EQ,C' '),
PUSH=(101:ID=8)),
IFTHEN=(WHEN=INIT,
OVERLAY=(109:C'A'))
//*
//JNF2CNTL DD *
INREC IFTHEN=(WHEN=GROUP,BEGIN=(66,1,CH,EQ,C' '),
PUSH=(101:ID=8)),
IFTHEN=(WHEN=(48,2,CH,EQ,C'!^'),
OVERLAY=(109:C'A'))
//*
SORTOUT had the same data as the 3rd sample above.
Joined: 17 Oct 2006 Posts: 2481 Location: @my desk
prino - You're welcome. That should be an easy change. I am not sure if the positions mentioned in the previous posts included RDW or not. But just note that the padding of work fields should be in the beginning for VB data sets. And there are parameters such as VLSHRT/VLSCMP to handle the short records.
It turns out that the problem is rather more complicated than I had expected!
The data contains groups (16 of them) of records that span more than one calendar day with the pseudo-interval, and are reduced to just one calendar day without it, and here no DFSORT will ever be able to make sense of removing the "!^" record as these groups have additional records where column 66 = "#" and an arrival date on the last of these, and I don't expect anyone can come up with a solution that moves some data (the time) back to the header of the group, and other data (a date) forward to the next group, eliminated the column 66="#" records and resets the day in the header (column 66 = blank) from "#" to the day of the first column 66 = "#" day.
Hell, even writing an edit macro to cater for these cases might take more time to debug, than to just convert them manually, which of course is not an option as that would mean keeping two versions of the input file for up to another 18-36+ months, when the oldest version to regression test falls off, a new version of the program is released about every six months to a year, depending on feature requests.
It will be an edit macro, if anyone is interested I'll post it here, although I don't expect anyone to have any use for it.
A Post Scriptum:This is historic data. Ideally it would need to be converted to XML (and become hugely bloated in the process), but for that to happen the community would have to jump in and define a DTD (or anything newer and whiter) and all the other relevant bits. I've asked the community a couple of times over the years, and the response? What response?
Joined: 17 Oct 2006 Posts: 2481 Location: @my desk
prino wrote:
these groups have additional records where column 66 = "#" and an arrival date on the last of these, and I don't expect anyone can come up with a solution that moves some data (the time) back to the header of the group, and other data (a date) forward to the next group, eliminated the column 66="#" records and resets the day in the header (column 66 = blank) from "#" to the day of the first column 66 = "#" day
Can you show an example (input and expected output) with the # records.
Started working on the edit macro, the one completed "WHEN" statement wipes out 320 of the 420 "!^" splits:
Code:
/*REXX */
"isredit macro"
"isredit x all"
"isredit f '!^' 48 first x"
do while rc = 0
"isredit (L) = line .zcsr"
"isredit (N) = linenum .zcsr"
"isredit label" n "= .L 0"
"isredit f '"substr(l, 1, 10)"' first x"
"isredit (H) = line .zcsr"
"isredit (N) = linenum .zcsr"
"isredit label" n "= .H 0"
select
/*-----------------------------------------------------------------+
| Easy-peasy, group doesn't contain multiple days, nor multiple |
| countries, so we can just paste the departure time over the |
| arrival time in the master, and comment out both the old master |
| and the now no longer required pseudo-split. |
+-----------------------------------------------------------------*/
when substr(h, 15, 1) \= '#' &,
substr(h, 54, 1) \= '*' then
do
new_arr = substr(l, 69, 5)
l = overlay('{--', l, 1)
l = overlay('%', l, 49)
nh = overlay(new_arr, h, 76)
h = overlay('{--', h, 1)
"isredit line_before .H = (NH)"
"isredit line .H = (H)"
"isredit line .L = (L)"
end
How did the '#' change to '2' - in the 3rd field. Is it taken from the '!^' record of that group? Or is it the 'dd' portion of the date from the last #-record?
How did the '#' change to '2' - in the 3rd field. Is it taken from the '!^' record of that group? Or is it the 'dd' portion of the date from the last #-record?
No, because there are no longer any day splits in this case, the 2 comes from the last daysplit before the one that's going to disappear, and given that there would, in this case, only one daysplit be left, that also gets deleted, and the '#' on the master that indicated that there were originally any daysplits is replaced by the last day. The day numbers are in essence nothing more than sequence numbers for the actual dates, and don't skip for missing actual dates, in other words:
would be perfectly acceptable if no activitiy took place on the intervening days.
The "! n" indicators are just sequence numbered splits, you could all turn them into "!42" and, I think, but I'm not 100% sure, the program could not care less.
Joined: 17 Oct 2006 Posts: 2481 Location: @my desk
The below will take care of moving the 23.56 from the last '!^' record to the 'group header' and also moving the date from the LAST '#' record to the header of the 'next group', assuming VB/259 input. The output matches with the expected output, except for one byte, the '#' changed to '2' in the expected output. Sorry I could not really comprehend from the explanation, which field/record in the input data goes as '2' in place of the '#' in the expected output.
EDIT : After re-reading one of the previous posts, I think I understand what you want. You need the day(3rd field) from the first # record (removed in output), in place of the #(3rd field) in the header record. It can be done. I'll try to post something later if I find some time.
Joined: 07 Feb 2009 Posts: 1306 Location: Vilnius, Lithuania
If there are only two daysplits (and I don't have any records with more), the '#' needs to come from the '2' in the same position in the penultimate record that has a '#' in column 66, in this case the first of the two.
Now just need to merge the two sets of sort control statements together, and I can compare the output with what my REXX exec (I finished it) produces.
BTW, all I get from the above, and yes, there are spaces in column 1:
Code:
1 //PRINOSOR JOB (PRINO), JOB07162
// 'RAHP SORT TRIAL', 00020000
// CLASS=A, 00030000
// MSGCLASS=H, 00040000
// MSGLEVEL=(2,0), 00050000
// NOTIFY=&SYSUID 00060000
//********************************************************************* 00090000
2 //PROCS JCLLIB ORDER=(PRINO.RAHP.PROC) 00100000
//********************************************************************* 00110000
3 //STEP001 EXEC PGM=SORT
4 //SYSOUT DD SYSOUT=*
5 //SORTJNF1 DD DISP=SHR,DSN=&SYSUID..RAHP.VEXT(WORKLIFT)
6 //SORTJNF2 DD DISP=SHR,DSN=&SYSUID..RAHP.VEXT(WORKLIFT)
7 //SORTOUT DD SYSOUT=*,DCB=*.SORTJNF1
8 //SYSIN DD *
9 //JNF1CNTL DD *
10 //JNF2CNTL DD *
ICH70001I PRINO LAST ACCESS AT 20:43:49 ON FRIDAY, FEBRUARY 22, 2019
IEF142I PRINOSOR STEP001 - STEP WAS EXECUTED - COND CODE 0016
IEF373I STEP/STEP001 /START 2019053.2047
IEF374I STEP/STEP001 /STOP 2019053.2047 CPU 0MIN 00.03SEC SRB 0MIN 00.00SEC VIRT 21
IEF375I JOB/PRINOSOR/START 2019053.2047
IEF376I JOB/PRINOSOR/STOP 2019053.2047 CPU 0MIN 00.03SEC SRB 0MIN 00.00SEC
1ICE143I 0 BLOCKSET COPY TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R10 - 20:47 ON FRI FEB 22, 2019 -
0 JOINKEYS FILE=F1,FIELDS=(5,9,A),SORTED,NOSEQCK
$
ICE005A 0 BLANK NEEDED IN COLUMN 1 OR OPERATION NOT DEFINED CORRECTLY
JOINKEYS FILE=F2,FIELDS=(5,9,A),SORTED,NOSEQCK
$
ICE005A 0 BLANK NEEDED IN COLUMN 1 OR OPERATION NOT DEFINED CORRECTLY
JOIN UNPAIRED,F1
$
ICE005A 0 BLANK NEEDED IN COLUMN 1 OR OPERATION NOT DEFINED CORRECTLY
REFORMAT FIELDS=(F1:1,4,?,F2:81,6,F1:5)
$
ICE005A 0 BLANK NEEDED IN COLUMN 1 OR OPERATION NOT DEFINED CORRECTLY
SORT FIELDS=COPY
OMIT COND=(78,2,CH,EQ,C'!^',OR,96,1,CH,EQ,C'#')
INREC IFTHEN=(WHEN=(96,1,CH,EQ,C' ',AND,5,1,CH,EQ,C'B'),
OVERLAY=(105:6,6),HIT=NEXT),
IFTHEN=(WHEN=(96,1,CH,EQ,C' '),
OVERLAY=(113:21,10))
OUTFIL BUILD=(1,4,31)
ICE056A 0 SORTIN NOT DEFINED
ICE751I 0 C5-K90014 C6-K90014 C7-BASE C8-K90014 E7-BASE
ICE052I 3 END OF DFSORT
Or is my version of DFSORT out of date?
If you want I an send you the original and macro massaged files in a ZIP (about 600K)
Joined: 07 Feb 2009 Posts: 1306 Location: Vilnius, Lithuania
The JCL is posted, it's a copy of what you posted, and there's no //SORTIN DD in that either? I'll see if the PM system here allows for an attachment of 582K. FWIW, don't worry that some very long URL's in "alt-worklift.dat" are truncated, I don't are about them on z/OS. "lift.dat" in the original input file. Both contain UTF-8 encoded data.
Joined: 07 Feb 2009 Posts: 1306 Location: Vilnius, Lithuania
I'll check the version of DFSORT later, but, unless you want to figure out if this conversion is actually possible, I'd suggest you don't spend too much more time on it, as I've managed to find a way, adding just two statements
Code:
/*----------------------------------------------------------+
| Save the "departure" time of the idle-time split |
+----------------------------------------------------------*/
if substr(owner -> list.cday, 3, 1) ^= '#' &
substr(list.type, 2, 1) = '^' then
pp_ntime = list.dtime;
and
Code:
/*------------------------------------------------------------+
| Use starting time of idle-time as end-of-ride time! |
+------------------------------------------------------------*/
if pp_ntime ^= -1 then
split_ruler(1).time(1) = pp_ntime;
else
to the program, to ignore 321 out of 420 of the "!^" idle pseudo-intervals, and I'm sure the other 99 will also follow RSN...
Removing the intervals, with all changes involved, from the input file turns out to lead to other problems that I'm still trying to figure out.
Anyway, I also applied the changes to two other versions of the program, written in Virtual Pascal, and in Virtual Pascal with more in-line assembler than actual Pascal, and the results match. A few quick manual checks also show that the new version is correct.
Joined: 07 Feb 2009 Posts: 1306 Location: Vilnius, Lithuania
Way behind on DFSORT maintenance,
Quote:
If you see:
ICE201I F RECORD TYPE ...
the F indicates you have the July, 2008 DFSORT functions (FINDREP, WHEN=GROUP, DATASORT, SUBSET, etc) and all of the earlier functions.
This function level corresponds to z/OS DFSORT V1R5 PTF UK90013 and z/OS DFSORT V1R10 PTF UK90014. You are behind on DFSORT functional PTFs. Ask your System Programmer to install z/OS DFSORT V1R10 PTF UK90025 or z/OS DFSORT V1R12 PTF UK90026.
And that's where I am, z/OS 1.10 forever...
Anyway, 417 out of 420 "!^" cases solved on-the-fly, will do the last three manually, and see if that has any influence.
Joined: 07 Feb 2009 Posts: 1306 Location: Vilnius, Lithuania
My final(?) post in this thread
The to-be-backed-out change in the input file format that is now not ever going to happen, was introduced in August 2009, and possibly even earlier.
I do know that at some stage there have been backports of minor changes to older versions of the program, just to allow the regression testing framework to run after minor positional changes in the format of the input file, and it may well have been that some of these changes masked out the introduction of the "idle time" pseudo splits. It would require restoring very old saved-as-xmit file versions (I've got about 220 of them, yes, really) to get to the bottom of this, but life's just way to short to indulge in this kind of "fun and games".
