Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

Algorithm to unpack ISPF PACKED format
Goto page 1, 2  Next
 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> TSO/ISPF
View previous topic :: :: View next topic  
Author Message
Robert Robinson

New User


Joined: 28 Sep 2012
Posts: 5
Location: Canada

PostPosted: Tue Oct 23, 2012 9:43 pm    Post subject: Algorithm to unpack ISPF PACKED format
Reply with quote

Hello - first time post!

I'm creating a tool that can download selected members of a PDS over FTP, some of which are in ISPF packed format. IBM's solution for this (http://www-01.ibm.com/support/docview.wss?uid=swg21111883) is to use 3.3 copy or LMCOPY prior to the FTP step. This is impractical due to the large number of members - we can't predict which will be downloaded. Also we can't avoid having them created in packed format because they are created by a vendor package. Finally, the end-user of the tool is not mainframe-savvy and don't even have TSO logon credentials, and can't manually unpack the member, so we are stuck with an FTP solution.

I have downloaded a packed file in binary EBCDIC and done some analysis and it looks like a kind of RLE encoding. I've decoded some of it but am getting stuck. Does anyone have the algorithm for decoding ISPF packed format?

Thanks,
Back to top
View user's profile Send private message

enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Tue Oct 23, 2012 9:58 pm    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

file 183 of the cbt tape has a packoff assembler program,
it does not look too difficult to port it to rexx

IMO the most practical solution would be to XMIT to a sequential dataset
FTP to the <linux/windows/....> client
UNXMIT the XMITTED file

search my posts for a working UNXMIT object REXX script
I am right now enhancing it to UNPACK ispf members
here is the link
http://www.ibmmainframes.com/viewtopic.php?t=59378&highlight=unxmit

the script posted is not the current one

ig You want I might upload the <CURRENT> updated/fixed(a couple of glitches) version
Back to top
View user's profile Send private message
Robert Robinson

New User


Joined: 28 Sep 2012
Posts: 5
Location: Canada

PostPosted: Tue Oct 23, 2012 10:36 pm    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

Thanks, Enrico, I was wondering if there was something in CBT that might do it. It's been a while since I did any BAL but I'll have a look.

Regarding your XMIT suggestion - it is a windows FTP client user that is initiating this transfer. Not sure how an FTP session can initiate the XMIT on the host. But if I use FTP from the client, if UNXMIT can handle the ISPF unpacking, I can translate the REXX to C# and incorporate it into my FTP client? Or call the Object REXX (I'd need to get the run-time environment for that...). So please, tweak away - much appreciated.

Thanks,
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Tue Oct 23, 2012 11:09 pm    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

Quote:
it is a windows FTP client user that is initiating this transfer.


1) things might be simple if You transfer a whole PDS

2) otherwise things get complicated by the member selection process

*) everything anyway depends on the <skills> of the end user

for case (1) the dumb solution could be
have a <script> which gets from user the dataset name

MKDIRs and CDS to the proper locaL path/directory

MGETs from the host everything in BINARY after that

a <script>/program to convert from ebcdic to ascii/text unpacking if necessary

pretty easy in REXX


the <demangling> will flow nicely for FIXED LENGHT records
a bit murkier for Variable length stuff

it is common consensus however that FTPeing in BINARY variable length records to process them on the PC
is a three stars pain

for (2) if also a member selection process has to take place,
the simplest way is to do all the work on the MF
and after that download everything with the proper ASCII translation

unless... mixed situation

have the user <manually>
CREATE/CD to the local directory
start the FTP session
CD to the remote directory
GET each single <member> as needed ( in BINARY )
quit the FTP session
invoke the <demangling> program

I feel that the whole requirement should be analyzed better
( maybe You have already done it,
but the info You have posted is not enough to provide a better advice )

just waiting for more details

for the plain UNXMIT part there are a few C programs hanging around
for example
http://cbttape.org/~jmorrison/
( the page uses frames, so You will have to click on downloads )

there is also a UNXMIT monster written in JAVA
( 52 MB against the meager 40KB of the REXX stuff ) icon_biggrin.gif
http://sourceforge.net/projects/unxmit/
Back to top
View user's profile Send private message
Robert Robinson

New User


Joined: 28 Sep 2012
Posts: 5
Location: Canada

PostPosted: Tue Oct 23, 2012 11:30 pm    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

Thank you very much for your thoughts, Enrico. I have gotten the PACKOFF member out of cbt file 183, and so I'm dusting off my 30-year-old assembler hat. I think I can figure out the unpacking procedure based on that.

To answer your questions,

1) Cannot transfer the whole PDS because it is very large (thousands of members), any of which might be written by this vendor process at any time.

2) The member selection process is hidden inside the Windows client I am writing - it knows the member name to download based on other data in the PC system.

My Windows client knows how to FTP Receive the PDS member - I am doing the FTP inside the Windows application, so the user isn't even aware of the FTP session. It was doing the file transfer in ASCII, and that was the problem, because the member is in ISPF packed format. So, now my application is doing the FTP Receive in Binary, and will be doing the the EBCDIC-ASCII translation internally. But first, what is missing is the <demangling> part - the ISPF unpacking.

But now that I have the PACKOFF source code courtesy of dear departed Gilbert Saint-Flour (respects), I think I can do that part. So thanks much for your consideration Enrico, I think I've got it now! And it will be interesting to read some assembler after all this time.

Regards,
Back to top
View user's profile Send private message
prino

Active Member


Joined: 07 Feb 2009
Posts: 984
Location: Oostende, Belgium

PostPosted: Wed Oct 24, 2012 2:22 am    Post subject:
Reply with quote

From the sadly defunct M V S Help site:

Code:
/* Rexx - Unpack */
/* This program unpacks ISPF Packed Files. These should
   have been downloaded as binary, so they will be in EBCDIC.*/
arg FileIn .
Parse var FileIn Filename '.' .     /* drop trailing extension */
FileOut = FileName'.unpacked'

AsciiNum = '272030313233343536373839'x      /* start with quote space */
AsciiUpp = '4142434445464748494A4B4C4D4E4F505152535455565758595A'x
AsciiLow = '6162636465666768696A6B6C6D6E6F707172737475767778797A'x
AsciiSpecial =  '2f2c2d2e3a3b28292a2b213d'x /* /,-.:;()*+|= */

EbcdicNUM = '7d40f0f1f2f3f4f5f6f7f8f9'x     /* start with quote space */
EbcdicUpp = 'c1c2c3c4c5c6c7c8c9d1d2d3d4d5d6d7d8d9e2e3e4e5e6e7e8e9'x
EbcdicLow = '818283848586878889919293949596979899a2a3a4a5a6a7a8a9'x
EbcdicSpecial = '616b604b7a5e4d5d5c4e4f7e'x

Ascii  = AsciiNum  || AsciiUpp  || AsciiLow  || AsciiSpecial
Ebcdic = EbcdicNum || EbcdicUpp || EbcdicLow || EbcdicSpecial

   /*
    |  ISO 8859-1 to CECP 1047 (Extended de-facto EBCDIC):
    */

   ebcdic =           '00010203372D2E2F1605250B0C0D0E0F'x      /* 00 */
   ebcdic = ebcdic || '101112133C3D322618193F271C1D1E1F'x      /* 10 */
   ebcdic = ebcdic || '405A7F7B5B6C507D4D5D5C4E6B604B61'x      /* 20 */
   ebcdic = ebcdic || 'F0F1F2F3F4F5F6F7F8F97A5E4C7E6E6F'x      /* 30 */
   ebcdic = ebcdic || '7CC1C2C3C4C5C6C7C8C9D1D2D3D4D5D6'x      /* 40 */
   ebcdic = ebcdic || 'D7D8D9E2E3E4E5E6E7E8E9ADE0BD5F6D'x      /* 50 */
   ebcdic = ebcdic || '79818283848586878889919293949596'x      /* 60 */
   ebcdic = ebcdic || '979899A2A3A4A5A6A7A8A9C04FD0A107'x      /* 70 */
   ebcdic = ebcdic || '202122232415061728292A2B2C090A1B'x      /* 80 */
   ebcdic = ebcdic || '30311A333435360838393A3B04143EFF'x      /* 90 */
   ebcdic = ebcdic || '41AA4AB19FB26AB5BBB49A8AB0CAAFBC'x      /* A0 */
   ebcdic = ebcdic || '908FEAFABEA0B6B39DDA9B8BB7B8B9AB'x      /* B0 */
   ebcdic = ebcdic || '6465626663679E687471727378757677'x      /* C0 */
   ebcdic = ebcdic || 'AC69EDEEEBEFECBF80FDFEFBFCBAAE59'x      /* D0 */
   ebcdic = ebcdic || '4445424643479C485451525358555657'x      /* E0 */
   ebcdic = ebcdic || '8C49CDCECBCFCCE170DDDEDBDC8D8EDF'x      /* F0 */

   /*
    | Hex table to aid in translating all 8-bit characters
    */

   ascii =          '000102030405060708090A0B0C0D0E0F'x         /* 00 */
   ascii = ascii || '101112131415161718191A1B1C1D1E1F'x         /* 10 */
   ascii = ascii || '202122232425262728292A2B2C2D2E2F'x         /* 20 */
   ascii = ascii || '303132333435363738393A3B3C3D3E3F'x         /* 30 */
   ascii = ascii || '404142434445464748494A4B4C4D4E4F'x         /* 40 */
   ascii = ascii || '505152535455565758595A5B5C5D5E5F'x         /* 50 */
   ascii = ascii || '606162636465666768696A6B6C6D6E6F'x         /* 60 */
   ascii = ascii || '707172737475767778797A7B7C7D7E7F'x         /* 70 */
   ascii = ascii || '808182838485868788898A8B8C8D8E8F'x         /* 80 */
   ascii = ascii || '909192939495969798999A9B9C9D9E9F'x         /* 90 */
   ascii = ascii || 'A0A1A2A3A4A5A6A7A8A9AAABACADAEAF'x         /* A0 */
   ascii = ascii || 'B0B1B2B3B4B5B6B7B8B9BABBBCBDBEBF'x         /* B0 */
   ascii = ascii || 'C0C1C2C3C4C5C6C7C8C9CACBCCCDCECF'x         /* C0 */
   ascii = ascii || 'D0D1D2D3D4D5D6D7D8D9DADBDCDDDEDF'x         /* D0 */
   ascii = ascii || 'E0E1E2E3E4E5E6E7E8E9EAEBECEDEEEF'x         /* E0 */
   ascii = ascii || 'F0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF'x         /* F0 */
   ascii = xrange()

signal on notready

x = charin(FileIn,2,2)               /* read 1st 2 chars */
if x \= '0140'x then
  do
    say FileIn 'does not contain the identifier 0140 - instead' c2x(x)
    say 'Length of data' length(x)
    exit 12
  end

/* start reading our way through the beast */
x = charin(FileIn,9,1)                    /* go to start of data */
y = ''                                     /* y is output line */
do forever
  select
    when x = 'ff'x then exit

    when x >= '80'x then
      do                   /* data following */
        Datalen = c2d(x) - 127
        x = charin(FileIn,,DataLen)
        y = y || translate(x,Ascii,Ebcdic)
        x = charin(FileIn)
      end

    when x = '7C'x then
      do
        rc = LineOut(FileOut,y)
        y = ''
        x = charin(FileIn)   /* skip this byte - record length */
        x = charin(FileIn)   /* get next field length byte */
      end

 /* 7e = 7a, but at the end of a maximum length logical record */
    when x = '7a'x |,       /* process repetition character */
         x = '7e'x then     /* process repetition character */
      do
        DataLen = c2d(charin(FileIn)) + 1
        RepChar = charin(FileIn)
        y = y || left('',DataLen,translate(RepChar,Ascii,Ebcdic))
        x = charin(FileIn)           /* Get next field length */
      end

    otherwise                /* spaces indicator */
      do
        NumSpaces = c2d(x) + 1
        y = y || left(' ',NumSpaces,' ')
        x = charin(FileIn)             /* Get next field length */
      end
    end
  end

notready:
exit
/*
00 01 40 E5 00 00 00 50

7E    Rep char
4F    79 (+1)
81    'a'

4E    78 (+1) spaces
FC    EOR Rep char
00    0 (+1)
82    'b'

4A    74 (+1) spaces
7E    Rep char
04    4 (+1)
83    'c'

7A    Rep char
1F    31 (+1)
84    'd'
7C    Rep char
2F    47 (+1)

80    128 - 127
F0    '0'
7C    Rep char
4E    78 (+1)

80    128 (- 127)
F1    '1'
7C    Rep char
4E    78 (+1)

99    153 (- 127)
81    'a'
82    'b'
.
.
a8    'y'
a9    'z'

7c    Rep char
35    53 (+1)

FF    eof
*/
/*
80a @ 1, 1b @ 80, 5c @ 76, 32d @ 1
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa


dddddddddddddddddddddddddddddddd
0
1
abcdefghijklmnopqrstuvwxyz
*/
/*
http://m v s h e l p.net/vbforums/showthread.php?
t=19766&highlight=translate+packed

I don't have any doc on the format, but it is a run length encoding
scheme. In theory, the byte that gets compressed out is variable and
is in the header, but in practice it is limited to spaces (x'40') in
the ISPF implementation. The CMS pack format used to be documented and
ISPF uses pretty much the same one except that there is modification
because CMS always stored packed data in 1024 byte records and ISPF
didn't have that luxury. There was an apar back in Version 2 days
(1985?) that modified the format slightly to deal with reblocking
during copies and I suspect that that is where the fluff you mentioned
comes from. I couldn't find the apars though, but I remember it was a
nightmare of an apar (1 for DM, 1 for PDF). If you can get a hold of
any ancient CMS publications, you may find the algorithm mostly there.
===
Well, I finally figured out what is going on, so here is the format of
ISPF Packed Data:

The first 4 bytes are, I think, a packed-format indicator
(X'000140E5'). Then there are three bytes of filler, and then a 1 byte
record length. After this, it starts getting interesting...

As nadel said, it is run-length encoded. The first byte in the string
is either a length field, or the special chars x'7C' or x'7A'.

7C is an endline indicator, and is followed by another 1 byte record
length.

7A is a repetition indicator - it will be followed by a length byte,
and then the repetition character.

All other fields are length indicators - if they are greater than or
equal to x'80' then they indicate length-x'80' bytes of data
following. If they are less than x'80' then they indicate that number
of spaces should be inserted.

To round it all off, all lengths are m.l., so they need to be
incremented by one if used in Rexx.
===
<BLOCKQUOTE>The first 4 bytes are, I think, a packed-format indicator
(X'000140E5')</BLOCKQUOTE> FYI, the fourth byte is the RECFM - in your
case the letter 'V' ( x'E5' ) for Variable. It could also be a 'F'
( x'C6' ) if the data is Fixed-Length. If the data is Fixed-length, no
endline indicators ( x'7C' ) will be present in the packed data, as the
end point of each record is fixed. It does appear that trailing spaces
in one record followed by leading spaces in the next are compressed
separately and not 'wrapped'.
===
The header actually contains 8 bytes --

X'000140',CL1'F or V',FL4'lrecl'

My personal guess is that true indicator is the first X'00'. The X'01'
may be the number of filler characters, and the X'40' is the first
(of 1) filler characters.

For example --

and

The first 8 bytes in the second example are the BDW and the first RDW
for a variable record data set.

If the unpacked source file contains variable length records, the
packed file also contains variable length records. Regardless of
record format, the recorded data "streams" over logical record
boundaries.
===
Another special character -- X'FF' = end of data
===
Two more comments --

- X'7C' (end of line for V format records). I am unable to deduce the
meaning of contents of the byte following the string definition.

- In the event a variable length line completely fills the LRECL,
there is no end of line indicator.
===
The following is the core part of a routine to expand ISPF packed
format.

In addition to previous notes --

X'FC' is a header followed by a text length-1 without the X'80' and
text. It seems to be mostly used at or near the end of a VB line.

If the actual line length is 1 less than the LRECL of a variable
length record, SPF adds a blank to the line. I retain the excess blank
in my output.

Code notes --

GETREC returns the first text byte in R7, the record length in R8,
and the last text byte in R9. It returns to +4 if not an EOF, to +0 if
an EOF.

PUTREC writes a logical record, 1st text byte in R0, the LRECL in R1

Not all the paths have been fully checked yet.
===
I've found another key -- X'7E. Same as X'7A', but at the end of a
maximum length logical record.

I have nearly completed a utility to transform a data set to its SPF
packed format. I have also updated the EXPAND utility to process the
new key I found, but it has not yet been tested.
*/
Back to top
View user's profile Send private message
Robert Robinson

New User


Joined: 28 Sep 2012
Posts: 5
Location: Canada

PostPosted: Wed Oct 24, 2012 5:40 am    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

Wow, that's great Prino - thanks a bunch!

Regards,
Back to top
View user's profile Send private message
mtaylor

Active User


Joined: 20 Feb 2009
Posts: 108
Location: Kansas City

PostPosted: Sat Oct 27, 2012 8:25 pm    Post subject:
Reply with quote

Please forgive the questions but I'm ignorant about this; what is ispf packed and where is it used? What types of files are these? What is the purpose of the windows application? Why does it need these files?
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Sat Oct 27, 2012 8:53 pm    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

Quote:
what is ispf packed and where is it used? What types of files are these? What is the purpose of the windows application? Why does it need these files?


1) it is the format used when using pack on while editing something under ISPF
2) any files editable by ISPF edit
3-4) irrelevant to the issue

it is usually transparent to ISPF interactive use ( warning is issued when switching )
but seldom people look at the ISPF warning messages when editing a member/file

being able to unpack ISPF packed files might be needed/nice to have when trying to process IDTF XMITted files on a PC
(
for example the CBT files ,
they contain lots of useful stuff, worth looking at to learn things
but sometimes not enough to load them to MF datasets
)

that' s what my unxmit REXX script does
the snippet posted does not support ISPF packed files/members
the one I am working on DOES.

PS.
if You are curious just create a member with an easy pattern,
PACK ON; SAVE

and look at it using DITTO
using ISPF browse/view/edit will just unpack under the covers ....

this is how it looks under ISPF
Code:

 EDIT       ENRICO.TEST.PDS(AAAAAAAA) - 01.01               Columns 00001 00072
 Command ===> res                                              Scroll ===> CSR
 ****** ***************************** Top of Data ******************************
 ==MSG> -CAUTION- Profile changed to PACK ON (from PACK OFF) because
 ==MSG>           data was in packed format.
 000001 AAAAAAAAAA00010000
 000002 AAAAAAAAAA00020000
 000003 AAAAAAAAAA00030000
 000004 AAAAAAAAAA00040000
 000005 AAAAAAAAAA00050000
 000006 AAAAAAAAAA00060000
 000007 AAAAAAAAAA00070000
 000008 AAAAAAAAAA00080000
 000009 AAAAAAAAAA00090000
 000010 AAAAAAAAAA00100000
 ****** **************************** Bottom of Data ******************************


the same in hex with DITTO

Code:
  Process   View   Options   Help
────────────────────────────────────────────────────────────────────────────────
 DITTO/ESA for MVS              DB - Disk Browse

 Command ===>                                                       Scroll PAGE

 CYL-HD-REC 000000084 01 003     Extent 1   of 1       Col 1        Format UPDN
 Volume ****** **** 3390     DSNAME ENRICO.TEST.PDS
                                                           Search limit *
 Rec Type  Len      1...5...10....5...20....5...30....5...40....5...50....5...60
 3  DATA   160 CHAR .. F...&:.A:.0.1:.0.:.A:.0.2:.0.:.A:.0.3:.0.:.A:.0.4:.0.:.A:
               ZONE 004C000570C70F8F70F370C70F8F70F370C70F8F70F370C70F8F70F370C7
               NUMR 01060000A91A2001A30DA91A2002A30DA91A2003A30DA91A2004A30DA91A
                    1...5...10....5...20....5...30....5...40....5...50....5...60

               CHAR .0.5:.0.:.A:.0.6:.0.:.A:.0.7:.0.:.A:.0.8:.0.:.A:.0.9:.0.:.Ab
               ZONE 0F8F70F370C70F8F70F370C70F8F70F370C70F8F70F370C70F8F70F370C8
               NUMR 2005A30DA91A2006A30DA91A2007A30DA91A2008A30DA91A2009A30DA912
                   61...5...70....5...80....5...90....5...00....5...10....5...20

               CHAR 001:.0..
               ZONE FFF70F3F44444444444444444444444444444444
               NUMR 001A40DF00000000000000000000000000000000
                  121...5...30....5...40....5...50....5...60
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Fri Nov 02, 2012 2:33 am    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

unfortunately the REXX snippet posted is incomplete....

it does not take into account the 79, 7D, F8, f9 control chars

also the comment
Quote:
/* skip this byte - record length */

denotes poor coding ...

it is the <residual> byte count ...
it useful to check for proper unpack process ...
for a VB
the length of the <accumulated> output buffer + the residual count + 1
will give the LRECL

for fixed length records ...
the length of the <accumulated> output buffer + the length of the filler + 1
... same as above
Back to top
View user's profile Send private message
prino

Active Member


Joined: 07 Feb 2009
Posts: 984
Location: Oostende, Belgium

PostPosted: Fri Nov 02, 2012 2:46 am    Post subject: Re: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

enrico-sorichetti wrote:
unfortunately the REXX snippet posted is incomplete....

it does not take into account the 79, 7D, F8, f9 control chars

also the comment
Quote:
/* skip this byte - record length */

denotes poor coding ...

That's what I saved from MVS help, so it might be that I missed subsequent posts.

So what do the missing control characters do?
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Fri Nov 02, 2012 5:00 am    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

just decoding by testing ...

PS seem OK,

testing PDS/PDSE
the CBTTAPE has some files with ISPF packed data ( simple format )

I was able to unxmit EVERYTHING, included the nested .XMIs

the snippet posted seems to work for lrecl 80 things
probably up to 256

here is the code snippet that unpacks

I started using the IF then ELSE and I left it that way ...
I will use a select when I polish everything up

Code:
do  forever
    if  olen = XRECL then do
        if  trim then ,
            obuf = strip(obuf,"T")
        call lineout outf, obuf
        outk += 1; obyt += length(obuf)
        obuf = ""; olen = 0
    end

    if  xbuf = "" then do
        say "oh shit ! empty unpack buffer"
        signal logic_error
    end

    xchr = substr(xbuf,1,1)

    if  xchr = "ff"x then ,
        leave

    if  xchr = "78"x  | xchr = "7c"x then do
        if  length(xbuf) <= 2 then do
            shft = 1
            leave
        end
        xlen = c2d(substr(xbuf,2,1)) + 1
        if  ( olen + xlen ) \= XRECL then do
            say "oh shit ! XRECL mismatch "
            signal logic_error
        end
        if  trim then ,
            obuf = strip(obuf,"T")
        call lineout outf, obuf
        outk += 1; obyt += length(obuf)
        obuf = ""; olen = 0
        xbuf = substr(xbuf, 3)
        iterate
    end
    else ,
    if  xchr = "79"x | xchr = "7d"x then do
        if  length(xbuf) <= 3 then do
            shft = 1
            leave
        end
        xlen = c2d(substr(xbuf,2,2)) + 1
        if  ( olen + xlen ) \= XRECL then do
            say "oh shit ! XRECL mismatch "
            signal logic_error
        end
        if  trim then ,
            obuf = strip(obuf,"T")
        call lineout outf, obuf
        outk += 1; obyt += length(obuf)
        obuf = ""; olen = 0
        xbuf = substr(xbuf, 4)
        iterate
    end
    else ,
    if  xchr = "7a"x | xchr = "7e"x then do
        xlen =  c2d(substr(xbuf, 2, 1)) + 1
        if  length(xbuf) <= 3 then do
            shft = 1
            leave
        end
        char = _e2a(substr(xbuf, 3, 1))
        obuf = obuf || copies(char, xlen)
        olen += xlen
        xbuf = substr(xbuf, 4)
        iterate
    end
    else ,
    if  xchr = "f8"x | xchr = "fc"x then do
        if  length(xbuf) <= 2 then do
            shft = 1
            leave
        end
        xlen = c2d(substr(xbuf,2,1)) + 1
        if  length(xbuf) <= xlen + 2 then do
            shft = 1
            leave
        end
        obuf = obuf || _e2a(substr(xbuf,3,xlen))
        olen += xlen
        xbuf = substr(xbuf, xlen + 3 )
        iterate
    end
    else ,
    if  xchr = "f9"x then do    /* most probably also "fd"x */
        if  length(xbuf) <= 3 then do
            shft = 1
            leave
        end
        xlen = c2d(substr(xbuf,2,2)) + 1
        if  length(xbuf) <= xlen + 3 then do
            shft = 1
            leave
        end
        obuf = obuf || _e2a(substr(xbuf, 4, xlen))
        olen += xlen
        xbuf = substr(xbuf, xlen + 4 )
        iterate
    end
    else ,
    if  xchr >= "80"x then do
        xlen = c2d(xchr) - 127
        if  length(xbuf) <= xlen + 1 then do
            shft = 1
            leave
        end
        obuf = obuf || _e2a(substr(xbuf, 2, xlen))
        olen += xlen
        xbuf = substr(xbuf,xlen + 2)
        iterate
    end
    else do
        xlen = c2d(xchr) + 1
        if  ( olen + xlen ) \= XRECL then do
            say "oh shit ! XRECL mismatch "
            signal logic_error
        end
        obuf = obuf || copies(" ", xlen)
        olen += xlen
        if  trim then ,
            obuf = strip(obuf,"T")
        call lineout outf, obuf
        outk += 1; obyt += length(obuf)
        obuf = ""; olen = 0
        xbuf = substr(xbuf, 2)
        iterate
    end

    signal logic_error

end


the test on the output length is needed because wit fixed length record
completely filled with <data>

there is no <terminator> field

as You can see from here

Code:
unxmit   -        >>12345678901234567890123456789012345678901234567890123456789012345678901234567890<<
unxmit   -      1 >>{.. F...&:.0.1abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmn<<
unxmit   -        >>C004C000570FCF888888888999999999AAAAAAAA888888888999999999AAAAAAAA88888888899999<<
unxmit   -        >>001060000A40A1123456789123456789234567891234567891234567892345678912345678912345<<

unxmit   -     81 >>o<<
unxmit   -        >>9<<
unxmit   -        >>6<<

unxmit   -        >>12345678901234567890123456789012345678901234567890123456789012345678901234567890<<
unxmit   -      1 >>{pqrstuv:.0.2abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmno<<
unxmit   -        >>C999AAAA70FCF888888888999999999AAAAAAAA888888888999999999AAAAAAAA888888888999999<<
unxmit   -        >>07892345A40A21234567891234567892345678912345678912345678923456789123456789123456<<

unxmit   -     81 >>p<<
unxmit   -        >>9<<
unxmit   -        >>7<<

unxmit   -        >>12345678901234567890123456789012345678901234567890123456789012345678901234567890<<
unxmit   -      1 >>{qrstuv:.0.3abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnop<<
unxmit   -        >>C99AAAA70FCF888888888999999999AAAAAAAA888888888999999999AAAAAAAA8888888889999999<<
unxmit   -        >>0892345A40A312345678912345678923456789123456789123456789234567891234567891234567<<


if You are interested I will post the whole shebang when finished
I am working also on the XMIT counterpart

cheers
Back to top
View user's profile Send private message
Phrzby Phil

Active Member


Joined: 31 Oct 2006
Posts: 961
Location: Richmond, Virginia

PostPosted: Fri Nov 02, 2012 5:29 pm    Post subject:
Reply with quote

Trying to think outside the (big) box:

For a non-technical solution - any chance for a negotiable sum of money the vendor can/will also send the changed members via another mode to your PC system?
Back to top
View user's profile Send private message
thlucas

New User


Joined: 27 Dec 2012
Posts: 6
Location: United States

PostPosted: Fri Jan 04, 2013 3:55 am    Post subject: C# Code to parse ISPF PACK ON Data
Reply with quote

Here's a C# function I developed recently that does what you want (I think). I've tested it against both FB and VB formatted files, with differeing LRECL's.

You will need to remove the "Trace" references, as it uses a SmartInspect implementation for tracing code execution / variables.

Hope it helps!

Code:

        /// <summary>
        /// ISPF PACK Control Flags enumeration.
        /// </summary>
        internal enum ISPFPackControlFlags : byte
        {
            /// <summary>
            /// 0x00 - Not set.
            /// </summary>
            None = 0x00,
            /// <summary>
            /// 0x78 - Repeating spaces with 1 byte length.
            /// Control byte is followed by 1 byte length.
            /// </summary>
            RepeatingSpacesLength1 = 0x78,
            /// <summary>
            /// 0x79 - Repeating spaces with 2 byte length.
            /// Control byte is followed by 2 byte length.
            /// </summary>
            RepeatingSpacesLength2 = 0x79,
            /// <summary>
            /// 0x7A - Repeating character with 1 byte length.
            /// Control byte is followed by 1 byte length, followed by character code to repeat.
            /// </summary>
            RepeatingCharacterLength1 = 0x7A,
            /// <summary>
            /// 0x7B - Repeating character with 2 byte length.
            /// Control byte is followed by 2 byte length, followed by character code to repeat.
            /// </summary>
            RepeatingCharacterLength2 = 0x7B,
            /// <summary>
            /// 0x7C - Repeating spaces with 1 byte length.
            /// Control byte is followed by 1 byte length.
            /// Occurs in varying length record format.
            /// </summary>
            RepeatingSpacesLength1V = 0x7C,
            /// <summary>
            /// 0x7D - Repeating spaces with 2 byte length.
            /// Control byte is followed by 2 byte length.
            /// Occurs in varying length record format.
            /// </summary>
            RepeatingSpacesLength2V = 0x7D,
            /// <summary>
            /// 0x7E - Repeating character with 1 byte length.
            /// Control byte is followed by 1 byte length, followed by character code to repeat.
            /// Occurs in varying length record format.
            /// </summary>
            RepeatingCharacterLength1V = 0x7E,
            /// <summary>
            /// 0x7F - Repeating character with 2 byte length.
            /// Control byte is followed by 2 byte length, followed by character code to repeat.
            /// Occurs in varying length record format.
            /// </summary>
            RepeatingCharacterLength2V = 0x7F,
            /// <summary>
            /// 0x80 - String with byte length in lower 7 bits of control flag.
            /// Control byte contains string length in bits 1-6, followed by string data.
            /// </summary>
            StringLengthControlMask = 0x80,
            /// <summary>
            /// 0xF8 - String with 1 byte length.
            /// Control byte is followed by 1 byte length, followed by string data.
            /// </summary>
            StringLength1 = 0xF8,
            /// <summary>
            /// 0xF9 - String with 2 byte length.
            /// Control byte is followed by 2 byte length, followed by string data.
            /// </summary>
            StringLength2 = 0xF9,
            /// <summary>
            /// 0xFC - String with 1 byte length.
            /// Control byte is followed by 1 byte length, followed by string data.
            /// Occurs in varying length record format.
            /// </summary>
            StringLength1V = 0xFC,
            /// <summary>
            /// 0xFD - String with 2 byte length.
            /// Control byte is followed by 2 byte length, followed by string data.
            /// Occurs in varying length record format.
            /// </summary>
            StringLength2V = 0xFD,
            /// <summary>
            /// 0xFF - indicates end of file.
            /// </summary>
            EndOfFile = 0xFF,
        }


        /// <summary>
        /// Unpacks a file that has been packed via ISPF PACK ON attribute.
        /// </summary>
        /// <param name="inputPath">Input path that contains ISPF EBCDIC packed data.</param>
        /// <param name="outputPath">Output path to write unpacked data to.</param>
        /// <param name="removeTrailingSpaces">True to remove trailing spaces from output records (ie Line-Sequential); otherwise, false to keep trailing spaces (ie Fixed-Length).</param>
        /// <param name="ebcdicEncoding">EBCDIC encoding used to convert EBCDIC data to ASCII text.</param>
        /// <param name="asciiEncoding">ASCII encoding used to convert EBCDIC data to ASCII text.</param>
        /// <param name="Trace">Trace session used to trace method execution.</param>
        /// <remarks>
        /// Packed ISPF member data should be transferred as BINARY content, which is what this
        /// method is expecting in the <paramref name="inputPath"/> parameter.
        /// <para>
        /// Removing trailing spaces will cause the resulting output file to be formatted as a
        /// line-sequential PC format; otherwise, the file will be formatted as a fixed-width
        /// PC format, with CRLF characters at the end of each line.
        /// </para>
        /// </remarks>
        /// <example>
        /// <include path='Methods/Method[@name="ISPFUnpack.1"]/Example/*' file='C3Dataset Docs.xml' />
        /// </example>
        public static void ISPFUnpack(string inputPath, string outputPath, bool removeTrailingSpaces, Encoding ebcdicEncoding, Encoding asciiEncoding, C3TraceSession Trace)
        {
            ISPFPackControlFlags flag = ISPFPackControlFlags.None;
            StreamReader sr = null;
            BinaryReader br = null;
            StreamWriter sw = null;
            string value = string.Empty;
            int reclen = 0;
            int strlen = 0;
            byte[] bytes;

            try
            {
                // ensure trace object is set.
                if (Trace == null)
                    Trace = C3Trace.Trace;

                // trace.
                Trace.EnterMethod(Level.Debug);
                if (Trace.IsOn(Level.Debug))
                {
                    Trace.LogDebug("ISPF unpack data file.");
                    Trace.LogValue(Level.Debug, "inputPath", inputPath);
                    Trace.LogValue(Level.Debug, "outputPath", outputPath);
                    Trace.LogValue(Level.Debug, "removeTrailingSpaces", removeTrailingSpaces);
                    Trace.LogValue(Level.Debug, "ebcdicEncoding", ebcdicEncoding);
                    Trace.LogValue(Level.Debug, "asciiEncoding", asciiEncoding);
                }

                // validations.
                if (inputPath == null) throw new ArgumentException("inputPath");
                if (outputPath == null) throw new ArgumentException("outputPath");

                // open binary EBCDIC file for input.
                sr = new StreamReader(inputPath);
                br = new BinaryReader(sr.BaseStream);

                // open ascii file for output.
                sw = new StreamWriter(outputPath);

                // trace.
                if (Trace.IsOn(Level.Debug))
                    Trace.LogBinaryFile(Level.Debug, "ISPF PACK ON Data File (EBCDIC)", inputPath);

                // ensure input file has ISPF PACK ON header bytes and logical record length.
                if (sr.BaseStream.Length < 8)
                    throw new Exception("Input file does not contain ISPF PACK ON header bytes!");

                // check input file for ISPF PACK ON header bytes (0x000140).
                bytes = br.ReadBytes(3);
                value = C3Convert.Hex(bytes);
                if (value != "000140")
                    throw new Exception("Input file does not contain ISPF PACK ON header bytes (0x000140)!");

                // trace.
                if (Trace.IsOn(Level.Debug))
                    Trace.LogValue(Level.Debug, "ISPF PACK ON Header Bytes detected", value);

                // get record format byte - (F)ixed or (V)ariable.
                bytes = br.ReadBytes(1);
                bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);

                // get record format string indicator (recfm).
                string recfm = asciiEncoding.GetString(bytes);

                // verify record format.
                if (recfm != "F" && recfm != "V")
                    throw new Exception("Record format is not (F)ixed or (V)ariable!");

                // trace.
                if (Trace.IsOn(Level.Debug))
                    Trace.LogValue(Level.Debug, "ISPF PACK ON Record Format", recfm);

                // skip hi order RDW bytes.
                bytes = br.ReadBytes(2);

                // read file record length (lrecl).
                bytes = br.ReadBytes(2);
                value = C3Convert.Hex(bytes);
                int lrecl = Convert.ToInt32(value, 16);
                StringBuilder sbRec = new StringBuilder();

                // trace.
                if (Trace.IsOn(Level.Debug))
                    Trace.LogValue(Level.Debug, "ISPF PACK ON Record Length", lrecl);

                // process all bytes.
                do
                {
                    // read indicator byte.
                    flag = (ISPFPackControlFlags)br.ReadByte();
                    value = C3Convert.Hex((byte)flag);

                    // end of file (ie 0xFF)?
                    if (flag == ISPFPackControlFlags.EndOfFile) break;

                    // process based on flag byte.
                    if (flag == ISPFPackControlFlags.RepeatingSpacesLength1) // 0x78
                    {
                        // control byte 0x78 = repeating spaces, 1 byte length.
                        // 0x78, followed by 1 byte length.
                        // occurs in fixed or varying record format.

                        // read length.
                        bytes = br.ReadBytes(1);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // convert to ascii and write to output file.
                        value = "".PadRight(strlen, ' ');
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.RepeatingSpacesLength2) // 0x79
                    {
                        // control byte 0x79 = repeating spaces, 2 byte length.
                        // 0x79, followed by 2 byte length.
                        // occurs in fixed or varying record format.

                        // read length.
                        bytes = br.ReadBytes(2);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // convert to ascii and write to output file.
                        value = "".PadRight(strlen, ' ');
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.RepeatingCharacterLength1) // 0x7A
                    {
                        // control byte 0x7A = repeating character, 1 byte length.
                        // 0x7A, followed by 1 byte length, followed by character code to repeat.
                        // occurs in fixed or varying record format.

                        // read length.
                        bytes = br.ReadBytes(1);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read repeating character.
                        bytes = br.ReadBytes(1);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = "".PadRight(strlen, Convert.ToChar(bytes[0]));
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.RepeatingCharacterLength2) // 0x7B
                    {
                        // control byte 0x7B = repeating character, 2 byte length.
                        // 0x7B, followed by 2 byte length, followed by character code to repeat.
                        // occurs in fixed or varying record format.

                        // read length.
                        bytes = br.ReadBytes(2);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read repeating character.
                        bytes = br.ReadBytes(1);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = "".PadRight(strlen, Convert.ToChar(bytes[0]));
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.RepeatingSpacesLength1V) // 0x7C
                    {
                        // control byte 0x7C = repeating spaces, 1 byte length.
                        // 0x7C, followed by 1 byte length.
                        // occurs in varying record format only.

                        // read length.
                        bytes = br.ReadBytes(1);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // convert to ascii and write to output file.
                        value = "".PadRight(strlen, ' ');
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.RepeatingSpacesLength2V) // 0x7D
                    {
                        // control byte 0x7D = repeating spaces, 2 byte length.
                        // 0x7D, followed by 2 byte length.
                        // occurs in varying record format only.

                        // read length.
                        bytes = br.ReadBytes(2);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // convert to ascii and write to output file.
                        value = "".PadRight(strlen, ' ');
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.RepeatingCharacterLength1V) // 0x7E
                    {
                        // control byte 0x7E = repeating character, 1 byte length.
                        // 0x7E, followed by 1 byte length, followed by character code to repeat.
                        // occurs in varying record format only.

                        // read length.
                        bytes = br.ReadBytes(1);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read repeating character.
                        bytes = br.ReadBytes(1);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = "".PadRight(strlen, Convert.ToChar(bytes[0]));
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.RepeatingCharacterLength2V) // 0x7F
                    {
                        // control byte 0x7F = repeating character, 2 byte length.
                        // 0x7F, followed by 2 byte length, followed by character code to repeat.
                        // occurs in varying record format only.

                        // read length.
                        bytes = br.ReadBytes(2);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read repeating character.
                        bytes = br.ReadBytes(1);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = "".PadRight(strlen, Convert.ToChar(bytes[0]));
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.StringLength1) // 0xF8
                    {
                        // control byte 0xF8 = string, 1 byte length.
                        // 0xF8, followed by 1 byte length, followed by string bytes.
                        // occurs in fixed or varying record format.

                        // read length.
                        bytes = br.ReadBytes(1);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read string.
                        bytes = br.ReadBytes(strlen);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = asciiEncoding.GetString(bytes);
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.StringLength2) // 0xF9
                    {
                        // control byte 0xF9 = string, 2 byte length.
                        // 0xF9, followed by 2 byte length, followed by string bytes.
                        // occurs in fixed or varying record format.

                        // read length.
                        bytes = br.ReadBytes(2);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read string.
                        bytes = br.ReadBytes(strlen);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = asciiEncoding.GetString(bytes);
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.StringLength1V) // 0xFC
                    {
                        // control byte 0xFC = string, 1 byte length.
                        // 0xFC, followed by 1 byte length, followed by string bytes.
                        // occurs in varying record format only.

                        // read length.
                        bytes = br.ReadBytes(1);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read string.
                        bytes = br.ReadBytes(strlen);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = asciiEncoding.GetString(bytes);
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if (flag == ISPFPackControlFlags.StringLength2V) // 0xFD
                    {
                        // control byte 0xFD = string, 2 byte length.
                        // 0xFD, followed by 2 byte length, followed by string bytes.
                        // occurs in varying record format only.

                        // read length.
                        bytes = br.ReadBytes(2);
                        value = C3Convert.Hex(bytes);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read string.
                        bytes = br.ReadBytes(strlen);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = asciiEncoding.GetString(bytes);
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else if ((flag & ISPFPackControlFlags.StringLengthControlMask) == ISPFPackControlFlags.StringLengthControlMask)
                    {
                        // control byte contains mask plus string length.
                        // control byte is followed by string bytes.

                        // drop string indicator from control byte, leaving length minus one.
                        value = C3Convert.Hex((byte)flag & 0x7F);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // read string.
                        bytes = br.ReadBytes(strlen);

                        // convert to ascii and write to output file.
                        bytes = Encoding.Convert(ebcdicEncoding, asciiEncoding, bytes);
                        value = asciiEncoding.GetString(bytes);
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }
                    else
                    {
                        // otherwise repeating spaces for strings less than 255 characters.

                        // control byte is the length.
                        value = C3Convert.Hex((byte)flag);
                        strlen = Convert.ToInt32(value, 16);  // get length minus one.
                        strlen = strlen + 1;                  // get length.

                        // format repeating spaces string.
                        value = "".PadRight(strlen, ' ');
                        sbRec.Append(value);

                        // add length to total length.
                        reclen = reclen + strlen;
                    }

                    // todo - test - should not happen, but pause if so.
                    if (reclen > lrecl)
                        reclen = reclen + 0;

                    // is this a complete record?
                    if (reclen == lrecl)
                    {
                        // write record, trimming ending spaces if necessary.
                        value = sbRec.ToString();
                        if (removeTrailingSpaces) value = value.TrimEnd();
                        sw.WriteLine(value);

                        // prepare for next record.
                        reclen = 0;
                        sbRec = new StringBuilder();
                    }

                } while (true);

                // close output file.
                sw.Flush();
                sw.Close();

                // trace.
                if (Trace.IsOn(Level.Debug))
                    Trace.LogTextFile(Level.Debug, "ISPF PACK OFF Data File (ASCII)", outputPath);
            }
            catch (C3Exception) { throw; } // pass-thru
            catch (Exception ex)
            {
                // handle unrecognized exceptions.
                throw new C3Exception(Trace, ex, C3Trace.CurrentMethodNameClass);
            }
            finally
            {
                // free resources.
                if (sr != null) sr.Dispose();
                if (sw != null) sw.Dispose();

                // trace.
                Trace.LeaveMethod(Level.Debug);
            }
        }



C# code to execute the method:

Code:

    try
    {
        // create encodings used to translate EBCDIC to ASCII.
        System.Text.Encoding ebcdic = System.Text.Encoding.GetEncoding("EBCDIC-CP-US");
        System.Text.Encoding ascii = System.Text.Encoding.GetEncoding("ISO8859-1");

        // unpack ISPF packed data, converting to line-sequential format.
        C3Dataset.ISPFUnpack(@"C:\C3\v9\Source\C3.Mvs\TestData\ISPFPack\PACKON\VB1024.dat", @"C:\C3\v9\Source\C3.Mvs\TestData\ISPFPack\UNPACKED\VB1024_CS.txt", true, ebcdic, ascii, C3Trace.Trace);
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.Message, C3Trace.CurrentMethodNameClass + " - " + ex.GetType().Name);
    }


VB.NET code to execute the method:

Code:

    Try
   
        ' create encodings used to translate EBCDIC to ASCII.
        Dim ebcdic As System.Text.Encoding = System.Text.Encoding.GetEncoding("EBCDIC-CP-US")
        Dim ascii As System.Text.Encoding = System.Text.Encoding.GetEncoding("ISO8859-1")

        ' unpack ISPF packed data, converting to line-sequential format.
        C3Dataset.ISPFUnpack("C:\C3\v9\Source\C3.Mvs\TestData\ISPFPack\PACKON\VB1024.dat", "C:\C3\v9\Source\C3.Mvs\TestData\ISPFPack\UNPACKED\VB1024_VB.txt", True, ebcdic, ascii, C3Trace.Trace)

    Catch ex As Exception

        MessageBox.Show(ex.Message, C3Trace.CurrentMethodNameClass & " - " & ex.GetType().Name)

    End Try
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Fri Jan 04, 2013 4:11 am    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

Hello and welcome to the forum,

Thank you for posting your code icon_smile.gif

d
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Fri Jan 04, 2013 5:58 am    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

for those interested ...

https://bitbucket.org/herc4mac/hercules-recipes
https://bitbucket.org/herc4mac/hercules-recipes/src

runs on all the platforms supported by open object rexx

here is the direct link to the unxmit readme
https://bitbucket.org/herc4mac/hercules-recipes/src/780e2a37d706747d4ca0993f3ccd5675e378becf/README.UNXMIT.txt?at=master
Back to top
View user's profile Send private message
thlucas

New User


Joined: 27 Dec 2012
Posts: 6
Location: United States

PostPosted: Sat Jan 05, 2013 7:18 pm    Post subject: Re: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

enrico-sorichetti wrote:
for those interested ...

here is the direct link to the unxmit readme
https://bitbucket.org/herc4mac/hercules-recipes/src/780e2a37d706747d4ca0993f3ccd5675e378becf/README.UNXMIT.txt?at=master


Hi Enrico - I read your README.UNXMIT.txt.

The problem with the CBTTAPE FILE872 you mentioned is that it uses a German EBCDIC codepage IBM273 for translation, as opposed to the "standard" EBCDIC codepage of CP01047.

I was having the same issue with my XMIT Analyzer for FILE872, in that all of the { and } characters were not displaying correctly. I contacted Roland Scholz (FILE872 author), who kindly instructed me to use the "IBM273" codepage for FILE872.

I also noticed that you were writing a Rexx to handle ISPF PACK ON data. I have a bunch of test files for different scenarios if you would like to test with - Fixed, Variable, differing LRECL's, etc. PM me, and I will ZIP them up and send them to you. I used these files to test my C# implementation of the ISPF PACK ON converter.

Hope it helps.
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Sat Jan 05, 2013 8:00 pm    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

Hi Todd,

thank You for offering

I already did some testing ( I guess it should be enough )

on PS/PDS/PDSE

fb 80,256,600,1024

vb 256,600,1024

in order to reverse engineer the whole stuff

PDS/PDSE with packed unpacked members

encapsulated XMIs and other binaries

did You notice that
the flags 78/79/tc/td appear only for the last field of the record
and for VB files they fill UP to the maximum record length

and used that in my script to check for lrecl mismatch

IMHO dealing with recfm U datasets on a PC really makes little sense

adding a new code page to my scripts is really easy
but
I am working on loading them dynamically
as soon I have finished I will update the README

processing VB partitioned dataset is a bit tricky
for non packed members
per ISPF behavior only the payload is there

for packed members following blindly the algorithm You will end up with blanka padded records

right now in my script the default is to trim trailing blanks
Back to top
View user's profile Send private message
thlucas

New User


Joined: 27 Dec 2012
Posts: 6
Location: United States

PostPosted: Sat Jan 05, 2013 10:31 pm    Post subject: Re: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

The files I mentioned are for testing the ISPF PACK ON processing - not for the XMIT. Though I do have a number of test files for XMIT's as well.

Quote:
did You notice that the flags 78/79/tc/td appear only for the last field of the record and for VB files they fill UP to the maximum record length.


Ya, it gets even tricker though - especially if you start rearranging the lines and input order. For example, try placing a repeating character as the first line of input and process it. Then try a non-repeating character as the first line of input and process it.

I went ahead and attached the test case files for ISPF PACK in case anyone needs it.
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Sun Jan 06, 2013 1:40 am    Post subject: Reply to: Algorithm to unpack ISPF PACKED format
Reply with quote

Quote:
For example, try placing a repeating character as the first line of input and process it.


nothing strange from my point of view

a PS 80 file shows as

footprint
Code:
000140C600000050

the x'50' is the LRECL

with one record of 80 a ( x'81' )
Code:
000140C6000000507A4F81FF

with two records
Code:
000140C6000000507A4F817A4F81FF


pretty normal

7A repetition flag
4F Length -1
81 the repeated chars

the sequence fills a 80 bytes buffer
at the next iteration since the length of the buffer is equal to the LRECL
the buffer is written and the waltz starts again

thats the logic I used

cheers
Enrico
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> TSO/ISPF All times are GMT + 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Match or compare two files in VB Format anatol DFSORT/ICETOOL 14 Thu Nov 03, 2016 7:41 pm
No new posts Low values Results from VARCHAR FORMAT balaji81_k DB2 10 Thu Oct 20, 2016 1:18 am
No new posts output data in CSV Format janmejay COBOL Programming 8 Sat Oct 15, 2016 2:20 pm
This topic is locked: you cannot edit posts or make replies. output data in CSV Format janmejay DFSORT/ICETOOL 3 Sat Oct 15, 2016 2:16 pm
No new posts ISPF program to set SCRNAME prino TSO/ISPF 5 Fri Sep 30, 2016 2:18 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us