Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

How are large files in a REXX program handled?

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> CLIST & REXX
View previous topic :: :: View next topic  
Author Message
sravantid

New User


Joined: 13 May 2008
Posts: 5
Location: hyderabad

PostPosted: Thu May 15, 2008 12:07 pm    Post subject: How are large files in a REXX program handled?
Reply with quote

I am working on a tool(based on REXX). The records are read from the input file and are displayed on the screen. The code is working fine when the input file contains few records, say 60,000. But when the no. of records exceed this limit(a file with 1 lakh records) the program abends.

Can anyone give me a solution for this?
Back to top
View user's profile Send private message

expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Thu May 15, 2008 12:09 pm    Post subject:
Reply with quote

Process them in batches of say 25,000 records per batch.

There are examples of doing this on the forum, because I posted one of them.
Back to top
View user's profile Send private message
sravantid

New User


Joined: 13 May 2008
Posts: 5
Location: hyderabad

PostPosted: Thu May 15, 2008 12:13 pm    Post subject:
Reply with quote

Do you mean to say that i have to split the file into smaller files and then process?
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Thu May 15, 2008 12:34 pm    Post subject:
Reply with quote

NO, not at all.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Thu May 15, 2008 12:39 pm    Post subject:
Reply with quote

Quote:

Do you mean to say that i have to split the file into smaller files and then process?

always easier to ask a question than look, isn't it?

no, just means that you read only a portion of the file. if you are using EXECIO, that can be controlled with variables containing the number of the 1st record to be read and the number of records to read.

if you I/O service is something else (cobol/asm pgm) then the service will need to be parameterized to only return 'so-many' records.

REXX is not really for large file management; large files need to processed in chunks.
Back to top
View user's profile Send private message
Pedro

Senior Member


Joined: 01 Sep 2006
Posts: 2025
Location: Silicon Valley

PostPosted: Thu May 15, 2008 10:30 pm    Post subject: Reply to: How are large files in a REXX program handled?
Reply with quote

> ... when the no. of records exceed this limit ... the program abends.

What kind of abend?

Can you use a bigger region size? It might just abend at a later time but you might be lucky in that it will allow your program to run successfully.
Back to top
View user's profile Send private message
cpuhawg

Active User


Joined: 14 Jun 2006
Posts: 331
Location: Jacksonville, FL

PostPosted: Fri May 16, 2008 12:46 am    Post subject: Reply to: How are large files in a REXX program handled?
Reply with quote

I have written dozen of REXX programs that support ISPF panels where the input is upwards of 600,000 records.

The problem is available storage in TSO to store the stem variables. REXX programs running in batch mode can obtain a greater amount of storage that ones running interactively in TSO with panels.

The trick is to CALL DFSORT/ICETOOL to isolate the records you need prior to using EXECIO to read them in. You can also reduce the record length using the OUTREC parameter to save on storage.

SORTWORK statements are allocated which allows the program to use userid prefixed datasets for sorting rather than TSO storage.

Here is some sample code:

Code:

USERD = USERID()       
USERD = STRIP(USERD,'B')
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SYSIN') PURGE"             
SYSDS = "('"||USERD||".QUAL2.SYSIN')"                             
ADDRESS TSO "FREE F(SYSIN)"                                         
ADDRESS TSO "ALLOC DA"||SYSDS||" F(SYSIN) NEW CATALOG UNIT(SYSDA)",
            "RECFM(F B) LRECL(80) BLKSIZE(27920) DSORG(PS)",       
            "TRACKS SPACE(1 1)"                                     
NEWSTACK                                                           
DD01 = " OPTION NOVLSCMP,VLSHRT  "                                 
QUEUE DD01                                                         
DD02 = " SORT FIELDS=(10,8,CH,A) "                                 
QUEUE DD02                                                         
DD03 = " INCLUDE COND=(10,2,CH,EQ,C'???')"                     
QUEUE DD03                                                         
DD04 = " OUTFIL OUTREC=(1:10,130),CONVERT     "                     
QUEUE DD04                                                         
QUECT = QUEUED()                                                   
"EXECIO "QUECT" DISKW SYSIN (FINIS"                                 
DELSTACK                                                           
ADDRESS TSO "FREE F(SORTOUT)"                                       
OUTDS = "('"||USERD||".QUAL2.OUTPUT')"                               
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.OUTPUT') PURGE"           
ADDRESS TSO "ALLOC DA"||OUTDS||" F(SORTOUT) NEW CATALOG",           
            "LRECL(130) BLKSIZE(27950) RECFM(F B) DSORG(PS)",       
            "TRACKS SPACE(50 25) UNIT(SYSDA)"                       
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SORTWK01') PURGE"         
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SORTWK02') PURGE"         
ADDRESS TSO "ALLOC DA('"||USERD||".QUAL2.SORTWK01') F(SORTWK01)", 
        "NEW CATALOG BLKSIZE(27998) RECFM(U) TRACKS SPACE(400 100)",
        "UNIT(SYSDA)"                                               
ADDRESS TSO "ALLOC DA('"||USERD||".QUAL2.SORTWK02') F(SORTWK02)", 
        "NEW CATALOG BLKSIZE(27998) RECFM(U) TRACKS SPACE(400 100)",
        "UNIT(SYSDA)"                                               
ADDRESS TSO "ALLOC F(SYSOUT) DUMMY"                                 
ADDRESS TSO "ALLOC F(SYSUDUMP) DUMMY"                               
ADDRESS TSO "ALLOC DA("||DATASET||") F(SORTIN) SHR REUSE"           
ADDRESS TSO "CALL 'SYS1.SORT.LIBRARY(SORT)' "                           
ADDRESS TSO "FREE FI(SORTWK01)"                                     
ADDRESS TSO "FREE FI(SORTWK02)"                                     
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SORTWK01') PURGE" 
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SORTWK02') PURGE" 
ADDRESS TSO "FREE F(SORTIN)"                               
ADDRESS TSO "FREE F(SORTOUT)"                               
ADDRESS TSO "FREE F(SYSOUT)"                               
ADDRESS TSO "FREE F(SYSUDUMP)"                             
ADDRESS TSO "FREE F(SYSIN)"                                 


You can still have storage problems if massive numbers of records are output from the sort, but if you are pulling 30 to 40 thousand records from a 600,000 record file, you should be OK.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Fri May 16, 2008 11:46 am    Post subject:
Reply with quote

This is what I was saying. Maybe the OP found it himself already icon_biggrin.gif
Code:

EOF = 0                               
"EXECIO 0 DISKR DCO (OPEN)"           
DO FOREVER UNTIL(EOF)                 
  "EXECIO 25000 DISKR DCO (STEM DCOI."
  IF RC <> 0 THEN EOF = 1             
  DO A = 1 TO DCOI.0                 
    *** PROCESS BATCH OF RECORDS *** 
  END                                 
END                                   
Back to top
View user's profile Send private message
rskumar

New User


Joined: 21 Nov 2005
Posts: 35
Location: chennai

PostPosted: Thu May 22, 2008 4:49 am    Post subject: Doubt on Expat's query
Reply with quote

Expat,

Why do you keep EOF as 0 in the beginning and read 0 records from the physical file? How does this help having continuity.

Actually I am writing a program to split the 60000 records of a file based on a condition and writing it into 2 files. I am facing an error "Unable to obtain storage" while reading the file into stem. Reading works for less number of records.

How to handle this?
Back to top
View user's profile Send private message
Pedro

Senior Member


Joined: 01 Sep 2006
Posts: 2025
Location: Silicon Valley

PostPosted: Thu May 22, 2008 5:55 am    Post subject: Reply to: How are large files in a REXX program handled?
Reply with quote

> Why do you keep EOF as 0 in the beginning...
EOF is just an arbitrary variable name used to control the end of the DO UNTIL loop. It will keep looping until the condition is true, so to start the loop, it has to be false. the UNTIL condition has to resolve to 0 or 1.


> read 0 records from the physical file?
The first EXECIO is just to open the file. I suspect you do not need it. I think the EXECIO within the loop will open the file if it is not already open. But the program should likely have an EXECIO after the loop to close the file.


> How does this help having continuity.
It sure of your question, but the first EXECIO in expat's example might be there for clarity.


Did you study expat's example from Fri May 16, 2008 11:16 am? Like several people have recommend:

1. you need to read a smaller group of the records, say 25000
2. then process that set of 25000 records
3. repeat steps 1 and 2 until done.

The last group of records will likely not be 25000, but DCOI.0 will have the actual count.
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Thu May 22, 2008 11:36 am    Post subject:
Reply with quote

Quote:

> read 0 records from the physical file?
The first EXECIO is just to open the file. I suspect you do not need it. I think the EXECIO within the loop will open the file it it is not already open. But the program should likely have an EXECIO after the loop to close the file.

Correct, I forgot to include the line
Code:

"EXECIO 0 DISKR DCO ( FINIS"

The reason I include these is to make sure that files are opened and closed, probably more important, properly because they can be processed in both online and batch. Online needs a little more care for closing and freeing files, so do that in batch too and it will always work icon_eek.gif sort of .......................
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> CLIST & REXX All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Executing OO COBOL program invoking J... Virendra Shambharkar COBOL Programming 2 Tue Jan 10, 2017 6:37 pm
No new posts Need Suggestion on COBOL program vickey_dw COBOL Programming 5 Thu Jan 05, 2017 10:55 pm
This topic is locked: you cannot edit posts or make replies. Calling REXX code from ISPF panel sanchita dey TSO/ISPF 4 Mon Jan 02, 2017 5:13 pm
No new posts REXX DB2: Dynamic allocation of DB2.D... BHAS CLIST & REXX 3 Mon Dec 19, 2016 8:26 pm
No new posts find particular member name in PDS us... ravi243 CLIST & REXX 10 Mon Dec 19, 2016 6:44 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us