|
View previous topic :: View next topic
|
| Author |
Message |
sravantid
New User
Joined: 13 May 2008
Posts: 5
Location: hyderabad
|
|
|
|
I am working on a tool(based on REXX). The records are read from the input file and are displayed on the screen. The code is working fine when the input file contains few records, say 60,000. But when the no. of records exceed this limit(a file with 1 lakh records) the program abends.
Can anyone give me a solution for this? |
|
| Back to top |
|
 |
expat
Global Moderator
Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales
|
|
|
|
Process them in batches of say 25,000 records per batch.
There are examples of doing this on the forum, because I posted one of them. |
|
| Back to top |
|
|
sravantid
New User
Joined: 13 May 2008
Posts: 5
Location: hyderabad
|
|
|
|
| Do you mean to say that i have to split the file into smaller files and then process? |
|
| Back to top |
|
|
expat
Global Moderator
Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales
|
|
|
|
| NO, not at all. |
|
| Back to top |
|
|
dbzTHEdinosauer
Global Moderator
Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne
|
|
|
|
| Quote: |
Do you mean to say that i have to split the file into smaller files and then process?
|
always easier to ask a question than look, isn't it?
no, just means that you read only a portion of the file. if you are using EXECIO, that can be controlled with variables containing the number of the 1st record to be read and the number of records to read.
if you I/O service is something else (cobol/asm pgm) then the service will need to be parameterized to only return 'so-many' records.
REXX is not really for large file management; large files need to processed in chunks. |
|
| Back to top |
|
|
Pedro
Global Moderator
Joined: 01 Sep 2006
Posts: 2547
Location: Silicon Valley
|
|
|
|
> ... when the no. of records exceed this limit ... the program abends.
What kind of abend?
Can you use a bigger region size? It might just abend at a later time but you might be lucky in that it will allow your program to run successfully. |
|
| Back to top |
|
|
cpuhawg
Active User
Joined: 14 Jun 2006
Posts: 331
Location: Jacksonville, FL
|
|
|
|
I have written dozen of REXX programs that support ISPF panels where the input is upwards of 600,000 records.
The problem is available storage in TSO to store the stem variables. REXX programs running in batch mode can obtain a greater amount of storage that ones running interactively in TSO with panels.
The trick is to CALL DFSORT/ICETOOL to isolate the records you need prior to using EXECIO to read them in. You can also reduce the record length using the OUTREC parameter to save on storage.
SORTWORK statements are allocated which allows the program to use userid prefixed datasets for sorting rather than TSO storage.
Here is some sample code:
| Code: |
USERD = USERID()
USERD = STRIP(USERD,'B')
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SYSIN') PURGE"
SYSDS = "('"||USERD||".QUAL2.SYSIN')"
ADDRESS TSO "FREE F(SYSIN)"
ADDRESS TSO "ALLOC DA"||SYSDS||" F(SYSIN) NEW CATALOG UNIT(SYSDA)",
"RECFM(F B) LRECL(80) BLKSIZE(27920) DSORG(PS)",
"TRACKS SPACE(1 1)"
NEWSTACK
DD01 = " OPTION NOVLSCMP,VLSHRT "
QUEUE DD01
DD02 = " SORT FIELDS=(10,8,CH,A) "
QUEUE DD02
DD03 = " INCLUDE COND=(10,2,CH,EQ,C'???')"
QUEUE DD03
DD04 = " OUTFIL OUTREC=(1:10,130),CONVERT "
QUEUE DD04
QUECT = QUEUED()
"EXECIO "QUECT" DISKW SYSIN (FINIS"
DELSTACK
ADDRESS TSO "FREE F(SORTOUT)"
OUTDS = "('"||USERD||".QUAL2.OUTPUT')"
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.OUTPUT') PURGE"
ADDRESS TSO "ALLOC DA"||OUTDS||" F(SORTOUT) NEW CATALOG",
"LRECL(130) BLKSIZE(27950) RECFM(F B) DSORG(PS)",
"TRACKS SPACE(50 25) UNIT(SYSDA)"
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SORTWK01') PURGE"
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SORTWK02') PURGE"
ADDRESS TSO "ALLOC DA('"||USERD||".QUAL2.SORTWK01') F(SORTWK01)",
"NEW CATALOG BLKSIZE(27998) RECFM(U) TRACKS SPACE(400 100)",
"UNIT(SYSDA)"
ADDRESS TSO "ALLOC DA('"||USERD||".QUAL2.SORTWK02') F(SORTWK02)",
"NEW CATALOG BLKSIZE(27998) RECFM(U) TRACKS SPACE(400 100)",
"UNIT(SYSDA)"
ADDRESS TSO "ALLOC F(SYSOUT) DUMMY"
ADDRESS TSO "ALLOC F(SYSUDUMP) DUMMY"
ADDRESS TSO "ALLOC DA("||DATASET||") F(SORTIN) SHR REUSE"
ADDRESS TSO "CALL 'SYS1.SORT.LIBRARY(SORT)' "
ADDRESS TSO "FREE FI(SORTWK01)"
ADDRESS TSO "FREE FI(SORTWK02)"
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SORTWK01') PURGE"
ADDRESS TSO "DELETE ('"||USERD||".QUAL2.SORTWK02') PURGE"
ADDRESS TSO "FREE F(SORTIN)"
ADDRESS TSO "FREE F(SORTOUT)"
ADDRESS TSO "FREE F(SYSOUT)"
ADDRESS TSO "FREE F(SYSUDUMP)"
ADDRESS TSO "FREE F(SYSIN)"
|
You can still have storage problems if massive numbers of records are output from the sort, but if you are pulling 30 to 40 thousand records from a 600,000 record file, you should be OK. |
|
| Back to top |
|
|
expat
Global Moderator
Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales
|
|
|
|
This is what I was saying. Maybe the OP found it himself already 
| Code: |
EOF = 0
"EXECIO 0 DISKR DCO (OPEN)"
DO FOREVER UNTIL(EOF)
"EXECIO 25000 DISKR DCO (STEM DCOI."
IF RC <> 0 THEN EOF = 1
DO A = 1 TO DCOI.0
*** PROCESS BATCH OF RECORDS ***
END
END
|
|
|
| Back to top |
|
|
rskumar
New User
Joined: 21 Nov 2005
Posts: 35
Location: chennai
|
|
|
|
Expat,
Why do you keep EOF as 0 in the beginning and read 0 records from the physical file? How does this help having continuity.
Actually I am writing a program to split the 60000 records of a file based on a condition and writing it into 2 files. I am facing an error "Unable to obtain storage" while reading the file into stem. Reading works for less number of records.
How to handle this? |
|
| Back to top |
|
|
Pedro
Global Moderator
Joined: 01 Sep 2006
Posts: 2547
Location: Silicon Valley
|
|
|
|
> Why do you keep EOF as 0 in the beginning...
EOF is just an arbitrary variable name used to control the end of the DO UNTIL loop. It will keep looping until the condition is true, so to start the loop, it has to be false. the UNTIL condition has to resolve to 0 or 1.
> read 0 records from the physical file?
The first EXECIO is just to open the file. I suspect you do not need it. I think the EXECIO within the loop will open the file if it is not already open. But the program should likely have an EXECIO after the loop to close the file.
> How does this help having continuity.
It sure of your question, but the first EXECIO in expat's example might be there for clarity.
Did you study expat's example from Fri May 16, 2008 11:16 am? Like several people have recommend:
1. you need to read a smaller group of the records, say 25000
2. then process that set of 25000 records
3. repeat steps 1 and 2 until done.
The last group of records will likely not be 25000, but DCOI.0 will have the actual count. |
|
| Back to top |
|
|
expat
Global Moderator
Joined: 14 Mar 2007
Posts: 8797
Location: Welsh Wales
|
|
|
|
| Quote: |
> read 0 records from the physical file?
The first EXECIO is just to open the file. I suspect you do not need it. I think the EXECIO within the loop will open the file it it is not already open. But the program should likely have an EXECIO after the loop to close the file. |
Correct, I forgot to include the line
| Code: |
"EXECIO 0 DISKR DCO ( FINIS"
|
The reason I include these is to make sure that files are opened and closed, probably more important, properly because they can be processed in both online and batch. Online needs a little more care for closing and freeing files, so do that in batch too and it will always work  sort of ....................... |
|
| Back to top |
|
|
|
|
