Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

File Recs across multiple "lines", want 1 complete

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL
View previous topic :: :: View next topic  
Author Message
John Del

New User


Joined: 27 Apr 2012
Posts: 24
Location: NY

PostPosted: Fri Nov 08, 2013 10:54 pm    Post subject: File Recs across multiple "lines", want 1 complete
Reply with quote

Hi -

Not quite sure how best to explain this scenario. Records may span across multple lines within a file, looking for a sort solution to have one complete logical record per line.

We have a couple of vendors in which we pick up FTP data from. From time to time, the files that they provide have the records as a single contiguous line, which ends up truncated to whatever or LOCSITE lrecl is set for. Setting an FTP command to NOTRUNC allows us to capture the transmission data into 80 byte, FB records with the data wrapped from line to line. The issue is how best to reconstruct this data into single logical records of one record per line within a file.

Getting the Vendors to change their method of creating the files have not been very successful.

The line/record CRLF characters may vary, say x'0A' or x'0D'. One option that has been suggested was to pull the file in binary type and use our encryption software to convert the data. That works fine, however, our sysprog has hinted that we may be changing our encryption software vendor in the near future. So I'm not too keen on that as a permanent solution.

We did develop a COBOL program that will handle formatting the input retrieved with the notrunc option into the desired output format. That's a working solution but obviously involves adding another step into the proc, as would the encryption software solution, too.

I accum all of our daily transmissions from these vendors into accum files as we pick up their files throughout the day. Therefore, I've been trying to figure out how to incorporate a solution into the sort step I use for the accum action as opposed to running a COBOL step for this.

I have made attempts but just quite don’t have the experience with DFSORT to figure it out to the end….. Is it possible to do this with sort?

Here is what I have as a start:


Code:
//STEP0001 EXEC PGM=SORT
//SYMNAMES DD DISP=SHR,DSN=USERLIB(SYMCONST)
//SYMNOUT  DD SYSOUT=*
//SYSOUT   DD SYSOUT=*
//SORTIN   DD *
~~
//         DD DISP=SHR,DSN=JMD.FTPDATA
//         DD *
~~
//         DD DISP=SHR,DSN=JMD.FTPDATA
//SORTOUT  DD SYSOUT=*
//SYSIN    DD *
  OPTION COPY

  INREC IFTHEN=(WHEN=GROUP,
     BEGIN=(1,2,CH,EQ,C'~~'),PUSH=(243:ID=1,SEQ=4)),
        IFTHEN=(WHEN=INIT,
         PARSE=(%01=(ENDBEFR=X'0A',FIXLEN=80),
                %02=(ENDBEFR=X'0A',FIXLEN=80),
                %03=(ENDBEFR=X'0A',FIXLEN=80)),
         BUILD=(%01,81:%02,162:%03,243:243,5))

/*


My various attempts at putting the data together in OUTRECS and OUTFIL have not been successful though, and my start as shown above may very well be flawed, too.

The typical input data has a minimum of 50 bytes and a max of 80 bytes coming to us. The data records can be varying lengths, but the input is FB 80. Number of records are typically a couple hundred; can be as many as just over a 1000.

Sample Input Data:
Code:
field1field2field3field4field5field6field7field8.ready1ready2ready3ready4ready5r
eady6ready7ready8.newrec1newrec2newrec3newrec4newrec5newrec6newrec7newrec8.anoth
er41another42another43another44another45another46another47another48.sample1sampl
e2sample3sample4sample5sample6sample7sample8.almost1almost2almost3almost4almost5
almost6almost7almost8.last71last72last73last74last75last76last77last78.     



Input Data, Hex displayed:
Code:

----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8
----+----F----+----F----+----F----+----F----+----F----+----F----+----F----+----F
----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8
 ------------------------------------------------------------------------------
field1field2field3field4field5field6field7field8.ready1ready2ready3ready4ready5r
88898F88898F88898F88898F88898F88898F88898F88898F09888AF9888AF9888AF9888AF9888AF9
695341695342695343695344695345695346695347695348A9514819514829514839514849514859
 ------------------------------------------------------------------------------
eady6ready7ready8.newrec1newrec2newrec3newrec4newrec5newrec6newrec7newrec8.anoth
888AF9888AF9888AF098A988F98A988F98A988F98A988F98A988F98A988F98A988F98A988F0899A8
51486951487951488A55695315569532556953355695345569535556953655695375569538A15638
 ------------------------------------------------------------------------------
er41another42another43another44another45another46another47another48.sample1sampl
89FF899A889FF899A889FF899A889FF899A889FF899A889FF899A889FF899A889FF0A89998FA8999
5941156385942156385943156385944156385945156385946156385947156385948A214735121473
 ------------------------------------------------------------------------------
e2sample3sample4sample5sample6sample7sample8.almost1almost2almost3almost4almost5
8FA89998FA89998FA89998FA89998FA89998FA89998F08999AAF8999AAF8999AAF8999AAF8999AAF
52214735321473542147355214735621473572147358A13462311346232134623313462341346235
   ISPF     -EDIT    EDIT2    DLIST    DLIST2   *DSLIST  SDSF2    ENDEVOR




Expected Output:
Code:
----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8
field1field2field3field4field5field6field7field8
ready1ready2ready3ready4ready5ready6ready7ready8
newrec1newrec2newrec3newrec4newrec5newrec6newrec7newrec8
another41another42another43another44another45another46another47another48
sample1sample2sample3sample4sample5sample6sample7sample8
almost1almost2almost3almost4almost5almost6almost7almost8
last71last72last73last74last75last76last77last78
******************************** Bottom of Data ********************************


Code:
ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1  (yay, finally up to H here!)



Ideas or pointers?

JD
Back to top
View user's profile Send private message

Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7308

PostPosted: Sat Nov 09, 2013 12:05 am    Post subject: Reply to: File Recs across multiple "lines", want
Reply with quote

PARSE and GROUP, yes. You're going to need to write up to two records per one input (if you are accurate with the minimum of 50, despite your sample data showing shortter) so you'll need BUILD on OUTFIL with the / (slash operator).

You shouldnt be able to get more than two end-of-records per record. An 80-byte record plus end-of-record is going to span two records with the end-of-record in byte one of the second. Another "special case" is where the end-of-record is in byte 80, in all other cases you have data to carry from one record to another (hence the GROUP).

Interesting task. Doable I think.
Back to top
View user's profile Send private message
John Del

New User


Joined: 27 Apr 2012
Posts: 24
Location: NY

PostPosted: Sat Nov 09, 2013 12:35 am    Post subject: Re: Reply to: File Recs across multiple "lines", w
Reply with quote

Bill Woodger wrote:
PARSE and GROUP, yes. You're going to need to write up to two records per one input (if you are accurate with the minimum of 50, despite your sample data showing shortter) so you'll need BUILD on OUTFIL with the / (slash operator).

You shouldnt be able to get more than two end-of-records per record. An 80-byte record plus end-of-record is going to span two records with the end-of-record in byte one of the second. Another "special case" is where the end-of-record is in byte 80, in all other cases you have data to carry from one record to another (hence the GROUP).

Interesting task. Doable I think.


Oops, sorry, I typed 50 for the minimum, I meant to type 40.

The actual record length is rarely ever less then 50 as the field elements are delimited in the files themselves with one required field being 23 bytes. Parsing for that is already handled further on in the cycle - as long as all records are one per line.

Thanks
Back to top
View user's profile Send private message
Skolusu

Senior Member


Joined: 07 Dec 2007
Posts: 2205
Location: San Jose

PostPosted: Mon Nov 11, 2013 11:59 pm    Post subject:
Reply with quote

John del,

It is easier to split the records based on the delimiter into separate records however the tricky part is the join the records which are split across records.

I am not sure as to why you need WHEN=GROUP in your original JCL as you are not using the ID anywhere.

Here is an ICETOOL JCL which will give you the desired results. I assumed you have a max of 6 records within a single record.

Code:

//STEP0100 EXEC PGM=ICETOOL                                           
//TOOLMSG  DD SYSOUT=*                                                 
//DFSMSG   DD SYSOUT=*                                                 
//IN       DD DISP=SHR,DSN=KOLUSU.JOHND.TEXT                           
//T1       DD DSN=&&T1,DISP=(,PASS),SPACE=(CYL,(1,1),RLSE)             
//OUT      DD SYSOUT=*                                                 
//TOOLIN   DD *                                                       
  RESIZE FROM(IN) TO(T1) TOLEN(81) USING(CTL1)                         
    COPY FROM(T1) TO(OUT) USING(CTL2)                                 
//CTL1CNTL DD *                                                       
  OPTION COPY                                                         
  INREC IFTHEN=(WHEN=INIT,                                             
        PARSE=(%01=(ENDAT=X'0A',FIXLEN=81),                           
               %02=(ENDAT=X'0A',FIXLEN=81),                           
               %03=(ENDAT=X'0A',FIXLEN=81),                           
               %04=(ENDAT=X'0A',FIXLEN=81),                           
               %05=(ENDAT=X'0A',FIXLEN=81),                           
               %06=(ENDAT=X'0A',FIXLEN=81)),                           
        BUILD=(%01,%02,%03,%04,%05,%06)),                             
  IFTHEN=(WHEN=(001,81,SS,NE,X'0A'),OVERLAY=(081:C'B'),HIT=NEXT),     
  IFTHEN=(WHEN=(082,81,SS,NE,X'0A'),OVERLAY=(162:C'B'),HIT=NEXT),     
  IFTHEN=(WHEN=(163,81,SS,NE,X'0A'),OVERLAY=(243:C'B'),HIT=NEXT),     
  IFTHEN=(WHEN=(244,81,SS,NE,X'0A'),OVERLAY=(324:C'B'),HIT=NEXT),     
  IFTHEN=(WHEN=(325,81,SS,NE,X'0A'),OVERLAY=(405:C'B'),HIT=NEXT),     
  IFTHEN=(WHEN=(406,81,SS,NE,X'0A'),OVERLAY=(486:C'B'),HIT=NEXT)       
                                                                       
  OUTFIL OMIT=(1,80,CH,EQ,C' '),                                       
  IFTHEN=(WHEN=GROUP,BEGIN=(81,1,CH,EQ,C'B'),PUSH=(82:1,80),RECORDS=2)
//*                                                                   
//CTL2CNTL DD *                                                       
  OMIT COND=(81,1,CH,EQ,C'B')                                         
  INREC IFOUTLEN=80,IFTHEN=(WHEN=INIT,BUILD=(82,80,1,80)),             
  IFTHEN=(WHEN=INIT,BUILD=(1,160,SQZ=(SHIFT=LEFT))),                   
  IFTHEN=(WHEN=INIT,FINDREP=(INOUT=(X'0A',C'')))                       
//*                                                                   


The output from this job is
Code:

field1field2field3field4field5field6field7field8                         
ready1ready2ready3ready4ready5ready6ready7ready8                         
newrec1newrec2newrec3newrec4newrec5newrec6newrec7newrec8                 
another41another42another43another44another45another46another47another48
sample1sample2sample3sample4sample5sample6sample7sample8                 
almost1almost2almost3almost4almost5almost6almost7almost8                 
last71last72last73last74last75last76last77last78                         
Back to top
View user's profile Send private message
John Del

New User


Joined: 27 Apr 2012
Posts: 24
Location: NY

PostPosted: Tue Nov 12, 2013 2:50 am    Post subject:
Reply with quote

Skolusu wrote:
John del,

It is easier to split the records based on the delimiter into separate records however the tricky part is the join the records which are split across records.

I am not sure as to why you need WHEN=GROUP in your original JCL as you are not using the ID anywhere.

Here is an ICETOOL JCL which will give you the desired results. I assumed you have a max of 6 records within a single record.



Hi Skolusu -

Excellent, thank you. I would not have thought of using RESIZE for this. I also need to remember how useful OVERLAY is and how to go about using it.

Since my original post, I had changed to using ICETOOL with a temp DSN and tinkered with the parameters. I separated the records in the 1st CNTL similar to as you have but I continued to fall down trying to get the records back together in my 2nd CNTL. My main issue was in how to reliably identify the single records from the others.

I see how and why you utilized OVERLAY in the first INREC to identify those records WITHOUT the char string, as opposed to those that do have it - which is what I had been doing. Utilizing OVERLAY as you did makes it less difficult to identify the single records that were contained wholly on one line and also in the grouping of those records with parts that spanned multiple records.



Note, my use of the WHEN=GROUP to assign an ID in the original post was from when I had tried concatenating the input file with itself to create a key using an ID that I could use for grouping later. For the JCL that I posted, it had no bearing on the rest of the parameters or the output. Sorry if it caused confusion.

Thanks - John
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> DFSORT/ICETOOL All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Data replication from multiple Db2 ta... kishpra DB2 5 Mon Mar 27, 2017 9:58 pm
No new posts Changing of LRECL of a file abdulrafi DFSORT/ICETOOL 2 Fri Mar 24, 2017 3:25 pm
No new posts Unable to catalog a gdg dataset resid... Shovan JCL & VSAM 7 Fri Mar 24, 2017 2:24 pm
No new posts splitting a file abdulrafi DFSORT/ICETOOL 4 Fri Mar 24, 2017 11:51 am
No new posts Receive a file using PCOMM macro Harald.v.K IBM Tools 0 Thu Mar 23, 2017 6:50 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us