I'm creating a tool that can download selected members of a PDS over FTP, some of which are in ISPF packed format. IBM's solution for this (http://www-01.ibm.com/support/docview.wss?uid=swg21111883) is to use 3.3 copy or LMCOPY prior to the FTP step. This is impractical due to the large number of members - we can't predict which will be downloaded. Also we can't avoid having them created in packed format because they are created by a vendor package. Finally, the end-user of the tool is not mainframe-savvy and don't even have TSO logon credentials, and can't manually unpack the member, so we are stuck with an FTP solution.
I have downloaded a packed file in binary EBCDIC and done some analysis and it looks like a kind of RLE encoding. I've decoded some of it but am getting stuck. Does anyone have the algorithm for decoding ISPF packed format?
Thanks, Enrico, I was wondering if there was something in CBT that might do it. It's been a while since I did any BAL but I'll have a look.
Regarding your XMIT suggestion - it is a windows FTP client user that is initiating this transfer. Not sure how an FTP session can initiate the XMIT on the host. But if I use FTP from the client, if UNXMIT can handle the ISPF unpacking, I can translate the REXX to C# and incorporate it into my FTP client? Or call the Object REXX (I'd need to get the run-time environment for that...). So please, tweak away - much appreciated.
it is a windows FTP client user that is initiating this transfer.
1) things might be simple if You transfer a whole PDS
2) otherwise things get complicated by the member selection process
*) everything anyway depends on the <skills> of the end user
for case (1) the dumb solution could be
have a <script> which gets from user the dataset name
MKDIRs and CDS to the proper locaL path/directory
MGETs from the host everything in BINARY after that
a <script>/program to convert from ebcdic to ascii/text unpacking if necessary
pretty easy in REXX
the <demangling> will flow nicely for FIXED LENGHT records
a bit murkier for Variable length stuff
it is common consensus however that FTPeing in BINARY variable length records to process them on the PC
is a three stars pain
for (2) if also a member selection process has to take place,
the simplest way is to do all the work on the MF
and after that download everything with the proper ASCII translation
unless... mixed situation
have the user <manually>
CREATE/CD to the local directory
start the FTP session
CD to the remote directory
GET each single <member> as needed ( in BINARY )
quit the FTP session
invoke the <demangling> program
I feel that the whole requirement should be analyzed better
( maybe You have already done it,
but the info You have posted is not enough to provide a better advice )
just waiting for more details
for the plain UNXMIT part there are a few C programs hanging around
for example
cbttape.org/~jmorrison/
( the page uses frames, so You will have to click on downloads )
there is also a UNXMIT monster written in JAVA
( 52 MB against the meager 40KB of the REXX stuff ) sourceforge.net/projects/unxmit/
Thank you very much for your thoughts, Enrico. I have gotten the PACKOFF member out of cbt file 183, and so I'm dusting off my 30-year-old assembler hat. I think I can figure out the unpacking procedure based on that.
To answer your questions,
1) Cannot transfer the whole PDS because it is very large (thousands of members), any of which might be written by this vendor process at any time.
2) The member selection process is hidden inside the Windows client I am writing - it knows the member name to download based on other data in the PC system.
My Windows client knows how to FTP Receive the PDS member - I am doing the FTP inside the Windows application, so the user isn't even aware of the FTP session. It was doing the file transfer in ASCII, and that was the problem, because the member is in ISPF packed format. So, now my application is doing the FTP Receive in Binary, and will be doing the the EBCDIC-ASCII translation internally. But first, what is missing is the <demangling> part - the ISPF unpacking.
But now that I have the PACKOFF source code courtesy of dear departed Gilbert Saint-Flour (respects), I think I can do that part. So thanks much for your consideration Enrico, I think I've got it now! And it will be interesting to read some assembler after all this time.
Joined: 07 Feb 2009 Posts: 1315 Location: Vilnius, Lithuania
From the sadly defunct M V S Help site:
Code:
/* Rexx - Unpack */
/* This program unpacks ISPF Packed Files. These should
have been downloaded as binary, so they will be in EBCDIC.*/
arg FileIn .
Parse var FileIn Filename '.' . /* drop trailing extension */
FileOut = FileName'.unpacked'
x = charin(FileIn,2,2) /* read 1st 2 chars */
if x \= '0140'x then
do
say FileIn 'does not contain the identifier 0140 - instead' c2x(x)
say 'Length of data' length(x)
exit 12
end
/* start reading our way through the beast */
x = charin(FileIn,9,1) /* go to start of data */
y = '' /* y is output line */
do forever
select
when x = 'ff'x then exit
when x >= '80'x then
do /* data following */
Datalen = c2d(x) - 127
x = charin(FileIn,,DataLen)
y = y || translate(x,Ascii,Ebcdic)
x = charin(FileIn)
end
when x = '7C'x then
do
rc = LineOut(FileOut,y)
y = ''
x = charin(FileIn) /* skip this byte - record length */
x = charin(FileIn) /* get next field length byte */
end
/* 7e = 7a, but at the end of a maximum length logical record */
when x = '7a'x |, /* process repetition character */
x = '7e'x then /* process repetition character */
do
DataLen = c2d(charin(FileIn)) + 1
RepChar = charin(FileIn)
y = y || left('',DataLen,translate(RepChar,Ascii,Ebcdic))
x = charin(FileIn) /* Get next field length */
end
otherwise /* spaces indicator */
do
NumSpaces = c2d(x) + 1
y = y || left(' ',NumSpaces,' ')
x = charin(FileIn) /* Get next field length */
end
end
end
notready:
exit
/*
00 01 40 E5 00 00 00 50
7E Rep char
4F 79 (+1)
81 'a'
4E 78 (+1) spaces
FC EOR Rep char
00 0 (+1)
82 'b'
dddddddddddddddddddddddddddddddd
0
1
abcdefghijklmnopqrstuvwxyz
*/
/*
http://m v s h e l p.net/vbforums/showthread.php?
t=19766&highlight=translate+packed
I don't have any doc on the format, but it is a run length encoding
scheme. In theory, the byte that gets compressed out is variable and
is in the header, but in practice it is limited to spaces (x'40') in
the ISPF implementation. The CMS pack format used to be documented and
ISPF uses pretty much the same one except that there is modification
because CMS always stored packed data in 1024 byte records and ISPF
didn't have that luxury. There was an apar back in Version 2 days
(1985?) that modified the format slightly to deal with reblocking
during copies and I suspect that that is where the fluff you mentioned
comes from. I couldn't find the apars though, but I remember it was a
nightmare of an apar (1 for DM, 1 for PDF). If you can get a hold of
any ancient CMS publications, you may find the algorithm mostly there.
===
Well, I finally figured out what is going on, so here is the format of
ISPF Packed Data:
The first 4 bytes are, I think, a packed-format indicator
(X'000140E5'). Then there are three bytes of filler, and then a 1 byte
record length. After this, it starts getting interesting...
As nadel said, it is run-length encoded. The first byte in the string
is either a length field, or the special chars x'7C' or x'7A'.
7C is an endline indicator, and is followed by another 1 byte record
length.
7A is a repetition indicator - it will be followed by a length byte,
and then the repetition character.
All other fields are length indicators - if they are greater than or
equal to x'80' then they indicate length-x'80' bytes of data
following. If they are less than x'80' then they indicate that number
of spaces should be inserted.
To round it all off, all lengths are m.l., so they need to be
incremented by one if used in Rexx.
===
<BLOCKQUOTE>The first 4 bytes are, I think, a packed-format indicator
(X'000140E5')</BLOCKQUOTE> FYI, the fourth byte is the RECFM - in your
case the letter 'V' ( x'E5' ) for Variable. It could also be a 'F'
( x'C6' ) if the data is Fixed-Length. If the data is Fixed-length, no
endline indicators ( x'7C' ) will be present in the packed data, as the
end point of each record is fixed. It does appear that trailing spaces
in one record followed by leading spaces in the next are compressed
separately and not 'wrapped'.
===
The header actually contains 8 bytes --
X'000140',CL1'F or V',FL4'lrecl'
My personal guess is that true indicator is the first X'00'. The X'01'
may be the number of filler characters, and the X'40' is the first
(of 1) filler characters.
For example --
and
The first 8 bytes in the second example are the BDW and the first RDW
for a variable record data set.
If the unpacked source file contains variable length records, the
packed file also contains variable length records. Regardless of
record format, the recorded data "streams" over logical record
boundaries.
===
Another special character -- X'FF' = end of data
===
Two more comments --
- X'7C' (end of line for V format records). I am unable to deduce the
meaning of contents of the byte following the string definition.
- In the event a variable length line completely fills the LRECL,
there is no end of line indicator.
===
The following is the core part of a routine to expand ISPF packed
format.
In addition to previous notes --
X'FC' is a header followed by a text length-1 without the X'80' and
text. It seems to be mostly used at or near the end of a VB line.
If the actual line length is 1 less than the LRECL of a variable
length record, SPF adds a blank to the line. I retain the excess blank
in my output.
Code notes --
GETREC returns the first text byte in R7, the record length in R8,
and the last text byte in R9. It returns to +4 if not an EOF, to +0 if
an EOF.
PUTREC writes a logical record, 1st text byte in R0, the LRECL in R1
Not all the paths have been fully checked yet.
===
I've found another key -- X'7E. Same as X'7A', but at the end of a
maximum length logical record.
I have nearly completed a utility to transform a data set to its SPF
packed format. I have also updated the EXPAND utility to process the
new key I found, but it has not yet been tested.
*/
Joined: 20 Feb 2009 Posts: 108 Location: Kansas City
Please forgive the questions but I'm ignorant about this; what is ispf packed and where is it used? What types of files are these? What is the purpose of the windows application? Why does it need these files?
what is ispf packed and where is it used? What types of files are these? What is the purpose of the windows application? Why does it need these files?
1) it is the format used when using pack on while editing something under ISPF
2) any files editable by ISPF edit
3-4) irrelevant to the issue
it is usually transparent to ISPF interactive use ( warning is issued when switching )
but seldom people look at the ISPF warning messages when editing a member/file
being able to unpack ISPF packed files might be needed/nice to have when trying to process IDTF XMITted files on a PC
(
for example the CBT files ,
they contain lots of useful stuff, worth looking at to learn things
but sometimes not enough to load them to MF datasets
)
that' s what my unxmit REXX script does
the snippet posted does not support ISPF packed files/members
the one I am working on DOES.
PS.
if You are curious just create a member with an easy pattern,
PACK ON; SAVE
and look at it using DITTO
using ISPF browse/view/edit will just unpack under the covers ....
this is how it looks under ISPF
Code:
EDIT ENRICO.TEST.PDS(AAAAAAAA) - 01.01 Columns 00001 00072
Command ===> res Scroll ===> CSR
****** ***************************** Top of Data ******************************
==MSG> -CAUTION- Profile changed to PACK ON (from PACK OFF) because
==MSG> data was in packed format.
000001 AAAAAAAAAA00010000
000002 AAAAAAAAAA00020000
000003 AAAAAAAAAA00030000
000004 AAAAAAAAAA00040000
000005 AAAAAAAAAA00050000
000006 AAAAAAAAAA00060000
000007 AAAAAAAAAA00070000
000008 AAAAAAAAAA00080000
000009 AAAAAAAAAA00090000
000010 AAAAAAAAAA00100000
****** **************************** Bottom of Data ******************************
the same in hex with DITTO
Code:
Process View Options Help
────────────────────────────────────────────────────────────────────────────────
DITTO/ESA for MVS DB - Disk Browse
Command ===> Scroll PAGE
CYL-HD-REC 000000084 01 003 Extent 1 of 1 Col 1 Format UPDN
Volume ****** **** 3390 DSNAME ENRICO.TEST.PDS
Search limit *
Rec Type Len 1...5...10....5...20....5...30....5...40....5...50....5...60
3 DATA 160 CHAR .. F...&:.A:.0.1:.0.:.A:.0.2:.0.:.A:.0.3:.0.:.A:.0.4:.0.:.A:
ZONE 004C000570C70F8F70F370C70F8F70F370C70F8F70F370C70F8F70F370C7
NUMR 01060000A91A2001A30DA91A2002A30DA91A2003A30DA91A2004A30DA91A
1...5...10....5...20....5...30....5...40....5...50....5...60
CHAR .0.5:.0.:.A:.0.6:.0.:.A:.0.7:.0.:.A:.0.8:.0.:.A:.0.9:.0.:.Ab
ZONE 0F8F70F370C70F8F70F370C70F8F70F370C70F8F70F370C70F8F70F370C8
NUMR 2005A30DA91A2006A30DA91A2007A30DA91A2008A30DA91A2009A30DA912
61...5...70....5...80....5...90....5...00....5...10....5...20
CHAR 001:.0..
ZONE FFF70F3F44444444444444444444444444444444
NUMR 001A40DF00000000000000000000000000000000
121...5...30....5...40....5...50....5...60
unfortunately the REXX snippet posted is incomplete....
it does not take into account the 79, 7D, F8, f9 control chars
also the comment
Quote:
/* skip this byte - record length */
denotes poor coding ...
it is the <residual> byte count ...
it useful to check for proper unpack process ...
for a VB
the length of the <accumulated> output buffer + the residual count + 1
will give the LRECL
for fixed length records ...
the length of the <accumulated> output buffer + the length of the filler + 1
... same as above
testing PDS/PDSE
the CBTTAPE has some files with ISPF packed data ( simple format )
I was able to unxmit EVERYTHING, included the nested .XMIs
the snippet posted seems to work for lrecl 80 things
probably up to 256
here is the code snippet that unpacks
I started using the IF then ELSE and I left it that way ...
I will use a select when I polish everything up
Code:
do forever
if olen = XRECL then do
if trim then ,
obuf = strip(obuf,"T")
call lineout outf, obuf
outk += 1; obyt += length(obuf)
obuf = ""; olen = 0
end
if xbuf = "" then do
say "oh shit ! empty unpack buffer"
signal logic_error
end
xchr = substr(xbuf,1,1)
if xchr = "ff"x then ,
leave
if xchr = "78"x | xchr = "7c"x then do
if length(xbuf) <= 2 then do
shft = 1
leave
end
xlen = c2d(substr(xbuf,2,1)) + 1
if ( olen + xlen ) \= XRECL then do
say "oh shit ! XRECL mismatch "
signal logic_error
end
if trim then ,
obuf = strip(obuf,"T")
call lineout outf, obuf
outk += 1; obyt += length(obuf)
obuf = ""; olen = 0
xbuf = substr(xbuf, 3)
iterate
end
else ,
if xchr = "79"x | xchr = "7d"x then do
if length(xbuf) <= 3 then do
shft = 1
leave
end
xlen = c2d(substr(xbuf,2,2)) + 1
if ( olen + xlen ) \= XRECL then do
say "oh shit ! XRECL mismatch "
signal logic_error
end
if trim then ,
obuf = strip(obuf,"T")
call lineout outf, obuf
outk += 1; obyt += length(obuf)
obuf = ""; olen = 0
xbuf = substr(xbuf, 4)
iterate
end
else ,
if xchr = "7a"x | xchr = "7e"x then do
xlen = c2d(substr(xbuf, 2, 1)) + 1
if length(xbuf) <= 3 then do
shft = 1
leave
end
char = _e2a(substr(xbuf, 3, 1))
obuf = obuf || copies(char, xlen)
olen += xlen
xbuf = substr(xbuf, 4)
iterate
end
else ,
if xchr = "f8"x | xchr = "fc"x then do
if length(xbuf) <= 2 then do
shft = 1
leave
end
xlen = c2d(substr(xbuf,2,1)) + 1
if length(xbuf) <= xlen + 2 then do
shft = 1
leave
end
obuf = obuf || _e2a(substr(xbuf,3,xlen))
olen += xlen
xbuf = substr(xbuf, xlen + 3 )
iterate
end
else ,
if xchr = "f9"x then do /* most probably also "fd"x */
if length(xbuf) <= 3 then do
shft = 1
leave
end
xlen = c2d(substr(xbuf,2,2)) + 1
if length(xbuf) <= xlen + 3 then do
shft = 1
leave
end
obuf = obuf || _e2a(substr(xbuf, 4, xlen))
olen += xlen
xbuf = substr(xbuf, xlen + 4 )
iterate
end
else ,
if xchr >= "80"x then do
xlen = c2d(xchr) - 127
if length(xbuf) <= xlen + 1 then do
shft = 1
leave
end
obuf = obuf || _e2a(substr(xbuf, 2, xlen))
olen += xlen
xbuf = substr(xbuf,xlen + 2)
iterate
end
else do
xlen = c2d(xchr) + 1
if ( olen + xlen ) \= XRECL then do
say "oh shit ! XRECL mismatch "
signal logic_error
end
obuf = obuf || copies(" ", xlen)
olen += xlen
if trim then ,
obuf = strip(obuf,"T")
call lineout outf, obuf
outk += 1; obyt += length(obuf)
obuf = ""; olen = 0
xbuf = substr(xbuf, 2)
iterate
end
signal logic_error
end
the test on the output length is needed because wit fixed length record
completely filled with <data>
Joined: 31 Oct 2006 Posts: 1050 Location: Richmond, Virginia
Trying to think outside the (big) box:
For a non-technical solution - any chance for a negotiable sum of money the vendor can/will also send the changed members via another mode to your PC system?
Joined: 27 Dec 2012 Posts: 6 Location: United States
Here's a C# function I developed recently that does what you want (I think). I've tested it against both FB and VB formatted files, with differeing LRECL's.
You will need to remove the "Trace" references, as it uses a SmartInspect implementation for tracing code execution / variables.
Hope it helps!
Code:
/// <summary>
/// ISPF PACK Control Flags enumeration.
/// </summary>
internal enum ISPFPackControlFlags : byte
{
/// <summary>
/// 0x00 - Not set.
/// </summary>
None = 0x00,
/// <summary>
/// 0x78 - Repeating spaces with 1 byte length.
/// Control byte is followed by 1 byte length.
/// </summary>
RepeatingSpacesLength1 = 0x78,
/// <summary>
/// 0x79 - Repeating spaces with 2 byte length.
/// Control byte is followed by 2 byte length.
/// </summary>
RepeatingSpacesLength2 = 0x79,
/// <summary>
/// 0x7A - Repeating character with 1 byte length.
/// Control byte is followed by 1 byte length, followed by character code to repeat.
/// </summary>
RepeatingCharacterLength1 = 0x7A,
/// <summary>
/// 0x7B - Repeating character with 2 byte length.
/// Control byte is followed by 2 byte length, followed by character code to repeat.
/// </summary>
RepeatingCharacterLength2 = 0x7B,
/// <summary>
/// 0x7C - Repeating spaces with 1 byte length.
/// Control byte is followed by 1 byte length.
/// Occurs in varying length record format.
/// </summary>
RepeatingSpacesLength1V = 0x7C,
/// <summary>
/// 0x7D - Repeating spaces with 2 byte length.
/// Control byte is followed by 2 byte length.
/// Occurs in varying length record format.
/// </summary>
RepeatingSpacesLength2V = 0x7D,
/// <summary>
/// 0x7E - Repeating character with 1 byte length.
/// Control byte is followed by 1 byte length, followed by character code to repeat.
/// Occurs in varying length record format.
/// </summary>
RepeatingCharacterLength1V = 0x7E,
/// <summary>
/// 0x7F - Repeating character with 2 byte length.
/// Control byte is followed by 2 byte length, followed by character code to repeat.
/// Occurs in varying length record format.
/// </summary>
RepeatingCharacterLength2V = 0x7F,
/// <summary>
/// 0x80 - String with byte length in lower 7 bits of control flag.
/// Control byte contains string length in bits 1-6, followed by string data.
/// </summary>
StringLengthControlMask = 0x80,
/// <summary>
/// 0xF8 - String with 1 byte length.
/// Control byte is followed by 1 byte length, followed by string data.
/// </summary>
StringLength1 = 0xF8,
/// <summary>
/// 0xF9 - String with 2 byte length.
/// Control byte is followed by 2 byte length, followed by string data.
/// </summary>
StringLength2 = 0xF9,
/// <summary>
/// 0xFC - String with 1 byte length.
/// Control byte is followed by 1 byte length, followed by string data.
/// Occurs in varying length record format.
/// </summary>
StringLength1V = 0xFC,
/// <summary>
/// 0xFD - String with 2 byte length.
/// Control byte is followed by 2 byte length, followed by string data.
/// Occurs in varying length record format.
/// </summary>
StringLength2V = 0xFD,
/// <summary>
/// 0xFF - indicates end of file.
/// </summary>
EndOfFile = 0xFF,
}
/// <summary>
/// Unpacks a file that has been packed via ISPF PACK ON attribute.
/// </summary>
/// <param name="inputPath">Input path that contains ISPF EBCDIC packed data.</param>
/// <param name="outputPath">Output path to write unpacked data to.</param>
/// <param name="removeTrailingSpaces">True to remove trailing spaces from output records (ie Line-Sequential); otherwise, false to keep trailing spaces (ie Fixed-Length).</param>
/// <param name="ebcdicEncoding">EBCDIC encoding used to convert EBCDIC data to ASCII text.</param>
/// <param name="asciiEncoding">ASCII encoding used to convert EBCDIC data to ASCII text.</param>
/// <param name="Trace">Trace session used to trace method execution.</param>
/// <remarks>
/// Packed ISPF member data should be transferred as BINARY content, which is what this
/// method is expecting in the <paramref name="inputPath"/> parameter.
/// <para>
/// Removing trailing spaces will cause the resulting output file to be formatted as a
/// line-sequential PC format; otherwise, the file will be formatted as a fixed-width
/// PC format, with CRLF characters at the end of each line.
/// </para>
/// </remarks>
/// <example>
/// <include path='Methods/Method[@name="ISPFUnpack.1"]/Example/*' file='C3Dataset Docs.xml' />
/// </example>
public static void ISPFUnpack(string inputPath, string outputPath, bool removeTrailingSpaces, Encoding ebcdicEncoding, Encoding asciiEncoding, C3TraceSession Trace)
{
ISPFPackControlFlags flag = ISPFPackControlFlags.None;
StreamReader sr = null;
BinaryReader br = null;
StreamWriter sw = null;
string value = string.Empty;
int reclen = 0;
int strlen = 0;
byte[] bytes;
try
{
// ensure trace object is set.
if (Trace == null)
Trace = C3Trace.Trace;
// validations.
if (inputPath == null) throw new ArgumentException("inputPath");
if (outputPath == null) throw new ArgumentException("outputPath");
// open binary EBCDIC file for input.
sr = new StreamReader(inputPath);
br = new BinaryReader(sr.BaseStream);
// open ascii file for output.
sw = new StreamWriter(outputPath);
// trace.
if (Trace.IsOn(Level.Debug))
Trace.LogBinaryFile(Level.Debug, "ISPF PACK ON Data File (EBCDIC)", inputPath);
// ensure input file has ISPF PACK ON header bytes and logical record length.
if (sr.BaseStream.Length < 8)
throw new Exception("Input file does not contain ISPF PACK ON header bytes!");
// check input file for ISPF PACK ON header bytes (0x000140).
bytes = br.ReadBytes(3);
value = C3Convert.Hex(bytes);
if (value != "000140")
throw new Exception("Input file does not contain ISPF PACK ON header bytes (0x000140)!");
// trace.
if (Trace.IsOn(Level.Debug))
Trace.LogValue(Level.Debug, "ISPF PACK ON Header Bytes detected", value);
// get record format byte - (F)ixed or (V)ariable.
bytes = br.ReadBytes(1);
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
// get record format string indicator (recfm).
string recfm = asciiEncoding.GetString(bytes);
// verify record format.
if (recfm != "F" && recfm != "V")
throw new Exception("Record format is not (F)ixed or (V)ariable!");
// trace.
if (Trace.IsOn(Level.Debug))
Trace.LogValue(Level.Debug, "ISPF PACK ON Record Format", recfm);
// skip hi order RDW bytes.
bytes = br.ReadBytes(2);
// read file record length (lrecl).
bytes = br.ReadBytes(2);
value = C3Convert.Hex(bytes);
int lrecl = Convert.ToInt32(value, 16);
StringBuilder sbRec = new StringBuilder();
// trace.
if (Trace.IsOn(Level.Debug))
Trace.LogValue(Level.Debug, "ISPF PACK ON Record Length", lrecl);
// process all bytes.
do
{
// read indicator byte.
flag = (ISPFPackControlFlags)br.ReadByte();
value = C3Convert.Hex((byte)flag);
// end of file (ie 0xFF)?
if (flag == ISPFPackControlFlags.EndOfFile) break;
// process based on flag byte.
if (flag == ISPFPackControlFlags.RepeatingSpacesLength1) // 0x78
{
// control byte 0x78 = repeating spaces, 1 byte length.
// 0x78, followed by 1 byte length.
// occurs in fixed or varying record format.
// read length.
bytes = br.ReadBytes(1);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// convert to ascii and write to output file.
value = "".PadRight(strlen, ' ');
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.RepeatingSpacesLength2) // 0x79
{
// control byte 0x79 = repeating spaces, 2 byte length.
// 0x79, followed by 2 byte length.
// occurs in fixed or varying record format.
// read length.
bytes = br.ReadBytes(2);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// convert to ascii and write to output file.
value = "".PadRight(strlen, ' ');
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.RepeatingCharacterLength1) // 0x7A
{
// control byte 0x7A = repeating character, 1 byte length.
// 0x7A, followed by 1 byte length, followed by character code to repeat.
// occurs in fixed or varying record format.
// read length.
bytes = br.ReadBytes(1);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = "".PadRight(strlen, Convert.ToChar(bytes[0]));
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.RepeatingCharacterLength2) // 0x7B
{
// control byte 0x7B = repeating character, 2 byte length.
// 0x7B, followed by 2 byte length, followed by character code to repeat.
// occurs in fixed or varying record format.
// read length.
bytes = br.ReadBytes(2);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = "".PadRight(strlen, Convert.ToChar(bytes[0]));
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.RepeatingSpacesLength1V) // 0x7C
{
// control byte 0x7C = repeating spaces, 1 byte length.
// 0x7C, followed by 1 byte length.
// occurs in varying record format only.
// read length.
bytes = br.ReadBytes(1);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// convert to ascii and write to output file.
value = "".PadRight(strlen, ' ');
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.RepeatingSpacesLength2V) // 0x7D
{
// control byte 0x7D = repeating spaces, 2 byte length.
// 0x7D, followed by 2 byte length.
// occurs in varying record format only.
// read length.
bytes = br.ReadBytes(2);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// convert to ascii and write to output file.
value = "".PadRight(strlen, ' ');
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.RepeatingCharacterLength1V) // 0x7E
{
// control byte 0x7E = repeating character, 1 byte length.
// 0x7E, followed by 1 byte length, followed by character code to repeat.
// occurs in varying record format only.
// read length.
bytes = br.ReadBytes(1);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = "".PadRight(strlen, Convert.ToChar(bytes[0]));
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.RepeatingCharacterLength2V) // 0x7F
{
// control byte 0x7F = repeating character, 2 byte length.
// 0x7F, followed by 2 byte length, followed by character code to repeat.
// occurs in varying record format only.
// read length.
bytes = br.ReadBytes(2);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = "".PadRight(strlen, Convert.ToChar(bytes[0]));
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.StringLength1) // 0xF8
{
// control byte 0xF8 = string, 1 byte length.
// 0xF8, followed by 1 byte length, followed by string bytes.
// occurs in fixed or varying record format.
// read length.
bytes = br.ReadBytes(1);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// read string.
bytes = br.ReadBytes(strlen);
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = asciiEncoding.GetString(bytes);
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.StringLength2) // 0xF9
{
// control byte 0xF9 = string, 2 byte length.
// 0xF9, followed by 2 byte length, followed by string bytes.
// occurs in fixed or varying record format.
// read length.
bytes = br.ReadBytes(2);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// read string.
bytes = br.ReadBytes(strlen);
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = asciiEncoding.GetString(bytes);
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.StringLength1V) // 0xFC
{
// control byte 0xFC = string, 1 byte length.
// 0xFC, followed by 1 byte length, followed by string bytes.
// occurs in varying record format only.
// read length.
bytes = br.ReadBytes(1);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// read string.
bytes = br.ReadBytes(strlen);
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = asciiEncoding.GetString(bytes);
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if (flag == ISPFPackControlFlags.StringLength2V) // 0xFD
{
// control byte 0xFD = string, 2 byte length.
// 0xFD, followed by 2 byte length, followed by string bytes.
// occurs in varying record format only.
// read length.
bytes = br.ReadBytes(2);
value = C3Convert.Hex(bytes);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// read string.
bytes = br.ReadBytes(strlen);
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = asciiEncoding.GetString(bytes);
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else if ((flag & ISPFPackControlFlags.StringLengthControlMask) == ISPFPackControlFlags.StringLengthControlMask)
{
// control byte contains mask plus string length.
// control byte is followed by string bytes.
// drop string indicator from control byte, leaving length minus one.
value = C3Convert.Hex((byte)flag & 0x7F);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// read string.
bytes = br.ReadBytes(strlen);
// convert to ascii and write to output file.
bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
value = asciiEncoding.GetString(bytes);
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
else
{
// otherwise repeating spaces for strings less than 255 characters.
// control byte is the length.
value = C3Convert.Hex((byte)flag);
strlen = Convert.ToInt32(value, 16); // get length minus one.
strlen = strlen + 1; // get length.
// format repeating spaces string.
value = "".PadRight(strlen, ' ');
sbRec.Append(value);
// add length to total length.
reclen = reclen + strlen;
}
// todo - test - should not happen, but pause if so.
if (reclen > lrecl)
reclen = reclen + 0;
// is this a complete record?
if (reclen == lrecl)
{
// write record, trimming ending spaces if necessary.
value = sbRec.ToString();
if (removeTrailingSpaces) value = value.TrimEnd();
sw.WriteLine(value);
// prepare for next record.
reclen = 0;
sbRec = new StringBuilder();
}
} while (true);
// close output file.
sw.Flush();
sw.Close();
// trace.
if (Trace.IsOn(Level.Debug))
Trace.LogTextFile(Level.Debug, "ISPF PACK OFF Data File (ASCII)", outputPath);
}
catch (C3Exception) { throw; } // pass-thru
catch (Exception ex)
{
// handle unrecognized exceptions.
throw new C3Exception(Trace, ex, C3Trace.CurrentMethodNameClass);
}
finally
{
// free resources.
if (sr != null) sr.Dispose();
if (sw != null) sw.Dispose();
// trace.
Trace.LeaveMethod(Level.Debug);
}
}
C# code to execute the method:
Code:
try
{
// create encodings used to translate EBCDIC to ASCII.
System.Text.Encoding ebcdic = System.Text.Encoding.GetEncoding("EBCDIC-CP-US");
System.Text.Encoding ascii = System.Text.Encoding.GetEncoding("ISO8859-1");
' create encodings used to translate EBCDIC to ASCII.
Dim ebcdic As System.Text.Encoding = System.Text.Encoding.GetEncoding("EBCDIC-CP-US")
Dim ascii As System.Text.Encoding = System.Text.Encoding.GetEncoding("ISO8859-1")
The problem with the CBTTAPE FILE872 you mentioned is that it uses a German EBCDIC codepage IBM273 for translation, as opposed to the "standard" EBCDIC codepage of CP01047.
I was having the same issue with my XMIT Analyzer for FILE872, in that all of the { and } characters were not displaying correctly. I contacted Roland Scholz (FILE872 author), who kindly instructed me to use the "IBM273" codepage for FILE872.
I also noticed that you were writing a Rexx to handle ISPF PACK ON data. I have a bunch of test files for different scenarios if you would like to test with - Fixed, Variable, differing LRECL's, etc. PM me, and I will ZIP them up and send them to you. I used these files to test my C# implementation of the ISPF PACK ON converter.
Joined: 27 Dec 2012 Posts: 6 Location: United States
The files I mentioned are for testing the ISPF PACK ON processing - not for the XMIT. Though I do have a number of test files for XMIT's as well.
Quote:
did You notice that the flags 78/79/tc/td appear only for the last field of the record and for VB files they fill UP to the maximum record length.
Ya, it gets even tricker though - especially if you start rearranging the lines and input order. For example, try placing a repeating character as the first line of input and process it. Then try a non-repeating character as the first line of input and process it.
I went ahead and attached the test case files for ISPF PACK in case anyone needs it.
For example, try placing a repeating character as the first line of input and process it.
nothing strange from my point of view
a PS 80 file shows as
footprint
Code:
000140C600000050
the x'50' is the LRECL
with one record of 80 a ( x'81' )
Code:
000140C6000000507A4F81FF
with two records
Code:
000140C6000000507A4F817A4F81FF
pretty normal
7A repetition flag
4F Length -1
81 the repeated chars
the sequence fills a 80 bytes buffer
at the next iteration since the length of the buffer is equal to the LRECL
the buffer is written and the waltz starts again
I must thank you all for most generously contributing to my little project - especially Enrico, Prino and thLucas.
I now have a functioning FTP downloader and unpacker integrated into my C# application. I ended up writing this from the comments in the assembler version that Enrico pointed me to in the CBT tape. It is limited in scope - it only handles variable length records up to 255 bytes and so doesn't handle any of the fixed-length control codes - but in my case I am always downloading a predictable LRECL/RECFM so it's Ok.
Please bear in mind then that this code only handles a subset of situations, but sufficient for my purposes. Thanks again for all your help, folks.
Here are the Unpack and Ebcdic conversion methods:
Code:
private StringBuilder Unpack(MemoryStream memStream, int readCount, char recFm, int lRecl)
{
byte[] upkTxt = new byte[255]; // One unpacked line.
int i = 0;
int j = 0;
int ll = 0;
byte ch = 0;
bool eol = false;
byte[] txt = new byte[readCount]; // Create a byte array and read stream into it.
memStream.Position = 0;
memStream.Read(txt, 0, readCount);
// Build an ASCII string - guess at 4x size of packed data.
StringBuilder sb = new StringBuilder(txt.Length * 4);
while (i < txt.Length)
{
byte eb = txt[i++]; // eb = the next Ebcdic packed Byte
if (eb == 0xff) // FF = EOF
break;
/* Cloned from Rexx Unpack from ***READ FORUM RULES***.NET board */
if (eb == 0xfc || eb == 0x7c) // End of line
{
if (eb == 0xfc) // FC has ll bytes of data following
{
ll = txt[i++];
for (int k = 0; k <= ll; k++) // Length count is 1 less than should be so <=
upkTxt[j++] = txt[i++];
}
if (eb == 0x7c)
i++; // Skipping over count of trailing blanks in line
eol = true;
}
else
if (eb == 0x7a || eb == 0x7e) // Repetitions of the next character
{
// 7e = but at the end of a maximum lngth logical record
ll = txt[i++];
ch = txt[i++];
for (int k = 0; k <= ll; k++) // Repeater count is 1 less than should be so <=
upkTxt[j++] = ch;
}
else
if (eb >= 0x80) // length = (value - 80 + 1) bytes of data following.
{
ll = eb - 0x80;
for (int k = 0; k <= ll; k++) // length count is 1 less than should be so <=
upkTxt[j++] = txt[i++];
}
else
{ // Repeat blanks.
ll = eb;
ch = 0x40;
for (int k = 0; k <= ll; k++) // Repeater count is 1 less than should be so <=
upkTxt[j++] = ch;
}
// Uncomment line below to have a look a the unpacked line so far.
// string txtString = ConvertEbcdicToAscii(upkTxt, 0, j < 256 ? j : 255);
if (eol || j >= lRecl) // End of line or we've gone over the LRECL.
{ // Append to StringBuilder and start new line.
sb.Append(ConvertEbcdicToAscii(upkTxt, 0, j) + Environment.NewLine);
Array.Clear(upkTxt, 0, 255);
j = 0;
eol = false;
}
}
return sb;
}
private string ConvertEbcdicToAscii(byte[] ebcdic_bytes, int index, int length)
{
char[] chars = new char[length];
EBCDICDecoder.GetChars(ebcdic_bytes, index, length, chars, 0);
return new String(chars);
}
Joined: 27 Dec 2012 Posts: 6 Location: United States
Sounds good Robert - as long as it fills the need at hand, then it's all good.
I moonlight writing code for a vendor (C-Cubed), so I had to account for other situations in my C# example that I posted. If you're interested, check out their web-site. It's full of C# friendly products that allow you to do quite a few things between your MVS systems and the Windows platform. We use their products quite extensively in the automation department at First Data Corp where I work. I'm not their pimp or anything - just like their products and support.
