Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

Buffer limit reached while handling huge records
Goto page 1, 2  Next
 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> COBOL Programming
View previous topic :: :: View next topic  
Author Message
rajendra kalepu

New User


Joined: 29 Jul 2011
Posts: 21
Location: India

PostPosted: Mon Mar 31, 2014 7:29 pm    Post subject: Buffer limit reached while handling huge records
Reply with quote

1. We receive a file which contains 3 types of records namely Type 0, Type A and Type B
2. All Type0, TypeA, Type contains a common key.
3. We are loading these three types in 3 individual arrays each having a limit of handling 5000 entries (i.e., occurs 5000)
Please find below example for better understanding

input file
0 AAA --> Type 0 Here AAA is the Key for below 3
1 AAA xyz abc --> Type 1
2 AAA def ghi --> Type 2
0 BBB BBB is the key for below 3
1 BBB ghi jkl
2 BBB jkl mno

So once we load Type0 , Type 1 and Type 2 records arrays for a particular Key 'AAA' , we will do some processing and Reset the internal table (assume emtying it)
We will do this for all keys until End of File is reached

Problem i am facing : Currently due to business expansion i am receiving close to 20k records per Key. I mean to say
0 AAA
1 AAA .....
1 AAA .....
:
:
:
10k assume
2 AAA .....
2 AAA .....
:
:
:
10k assume
Total 20k

2. We have an counters for each type (Type0, Type 1 and Type 2) which increments by one on loading the internal tables

3. Due to the exceeding of internal array size i am getting SOC7 at Counter varaible (On analysis i came to know that its because of exceeding array limit)

4. I even tried to increase the limit as a temporary fix but the temporary space allocated for the job is limited (getting an error " Buffer limit reached")
More over i just dont want to increase array limit because i feel its not a good coding practise

Please suggest me a possible solution for this scenario to handle any amount of records in coming future wiht out depending on array limit

It will be very helpful in giving a permanent fix to this issue

Thanks in advance.
Back to top
View user's profile Send private message

Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 1712
Location: UK

PostPosted: Mon Mar 31, 2014 7:41 pm    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

do you really need all the records for a key in storage at the same time? I.e. are you referring to records more than once? If so ,can you store only the firelds that you need? If not, then you do not need to store them simply read/process/read next
Back to top
View user's profile Send private message
Terry Heinze

JCL Moderator


Joined: 14 Jul 2008
Posts: 1238
Location: Richfield, MN, USA

PostPosted: Mon Mar 31, 2014 8:08 pm    Post subject:
Reply with quote

As Nic says, try to store only the needed fields of the records you're storing. The compiler limits are very large these days. See Appendix B in the COBOL Language Reference Manual.
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Mon Mar 31, 2014 8:23 pm    Post subject:
Reply with quote

Hello,

Quote:
We are loading these three types in 3 individual arrays each having a limit of handling 5000 entries (i.e., occurs 5000)
How was the 5000 determied? This is Not a very large number . . .
Quote:
More over i just dont want to increase array limit because i feel its not a good coding practise
Why do you believe more than 5000 is not good?

What is causing the "buffer limit" error?

If the input is "in order" why not summarize as data is read and eliminate the array(s). Probably there is something i misunderstand.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Mon Mar 31, 2014 9:00 pm    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

You need to be explicit about what "Buffer limit reached" means.

If a table needs to hold 20,000 entries, it needs 20,000 entries. In fact, it is good practice to have a growth projection and ensure you don't have to re-visit the same thing in the short/medium term.

You have 128MB of WORKING-STORAGE. How big are your records?

Enterprise COBOL V5.1 has "unbounded" tables (limit not imposed at compile time) but as others have indicated, do you need to hold all the storage anyway?
Back to top
View user's profile Send private message
Bill O'Boyle

CICS Moderator


Joined: 14 Jan 2008
Posts: 2502
Location: Atlanta, Georgia, USA

PostPosted: Mon Mar 31, 2014 10:16 pm    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

Bill,

We had Tom Ross of IBM here doing a COBOL 5.1 overview, so I asked him whether "Unbound" could take on the PICTURE clause maximum of the DEPENDING ON (which in my example, was an unsigned binary-fullword) and he said YES.

I never posted my VSAMXREF sub-program, which you CALL from COBOL and it returns the number of records on the target VSAM file, using a SHOWCB Macro.

HTH....
Back to top
View user's profile Send private message
Rohit Umarjikar

Senior Member


Joined: 21 Sep 2010
Posts: 1609
Location: NY,USA

PostPosted: Tue Apr 01, 2014 12:43 am    Post subject:
Reply with quote

You might wants to a multithreading here, split the input file depending upon the "type" , by this you would only have a single "Type" in each file where the records are now controlled and then you can process them parallely using scheduling and at last concatenate all output file into a single file.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Tue Apr 01, 2014 2:24 am    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

Bill,

I guess that makes sense, although it would be nice if the documentation of UNBOUNDED in the Language Reference was specific :-)

For the OCCURS DEPENDING ON:

Quote:
OCCURS DEPENDING ON clause
The OCCURS DEPENDING ON clause specifies variable-length tables.
data-name-1
Identifies the object of the OCCURS DEPENDING ON clause; that is, the
data item whose current value represents the current number of
occurrences of the subject item. The contents of items whose occurrence
numbers exceed the value of the object are undefined
.


Even that could be clearer.

So, UBOUNDED has no specific maximum, but is bounded by the data-name-1 that is mentioned in the OCCURS DEPENDING ON clause.
Back to top
View user's profile Send private message
rajendra kalepu

New User


Joined: 29 Jul 2011
Posts: 21
Location: India

PostPosted: Tue Apr 01, 2014 12:40 pm    Post subject:
Reply with quote

Hi All,

Thank you for your response

I will try all the possible solutions posted here and will post my results shortly

Thank and Regards
Rajendra
Back to top
View user's profile Send private message
rajendra kalepu

New User


Joined: 29 Jul 2011
Posts: 21
Location: India

PostPosted: Tue Apr 01, 2014 12:57 pm    Post subject:
Reply with quote

Hi All,

Currently we are using "OCCURS DEPENDING" clause only in the program for all the 3 tables.
Actually when i tried to increase the array limit to more than 10000 per table
i am receving the following error during compile time

'THE SIZE OF THE "WORKING-STORAGE SECTION" EXCEEDED THE LIMIT OF 16,777,215 BYTES ALLOWED UNDER THE "RENT" AND"DATA(24)" COMPILER OPTIONS. THE "DATA(31)" OPTION WAS ASSUMED'

The program was existing from past 15 years. We are facing this problem recently because of the business expansion

Please let me know for more information. Do the needful

Thanks and regards,
Rajendra
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Tue Apr 01, 2014 1:38 pm    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

Please include the entire error message, including the error reference which is at the start of the message.

Do you need DATA(24) for your program? With DATA(31) you will have 128MB of WORKING-STORAGE, enough for 20,000+ entries for each table.

If you need DATA(24), explain why.

If you need DATA(24) you'll have to look at the suggestions for cutting things down to size.
Back to top
View user's profile Send private message
rajendra kalepu

New User


Joined: 29 Jul 2011
Posts: 21
Location: India

PostPosted: Tue Apr 01, 2014 3:39 pm    Post subject:
Reply with quote

Hi Mistermind,

Thats great. It worked fine. I coded DATA(31) in my program and i could see the compilation went fine even though the array limit has been set to 20000 records per table.

But i faced problem during the run time. I got an error
"An attempt was made to pass a parameter address above 16megabytes to AMODE(24) sub program"

Unfortunately we are calling a Subprogram defined at AMODE(24) which is used by different applications in our area

Please help me in this regard

Thanks and Regards,
Rajendra
Back to top
View user's profile Send private message
Bill O'Boyle

CICS Moderator


Joined: 14 Jan 2008
Posts: 2502
Location: Atlanta, Georgia, USA

PostPosted: Tue Apr 01, 2014 4:24 pm    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

When a 31-Bit program calls a 24-Bit program, what you could do is write a LINKAGE ASSISTANT program, which is AMODE 31 but builds a 24-Bit Parmlist to be passed to the 24-Bit sub-program, basically as a Pass-Through.

This can be done easily in Assembler, using a GETMAIN or STORAGE OBTAIN Macro, with LOC=24 within the LINKAGE ASSISTANT program and using this Below-the-Line Storage as the 24-Bit Parmlist.

You may also want to look at LE Callable Service routine CEEGTST, but I'm not sure you can specify a particular AMODE in the LINKAGE ASSISTANT, because I believe CEEGTST takes on the characteristics of the Caller.

But, it's woth looking into.

HTH....
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Tue Apr 01, 2014 5:30 pm    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

Does the sub-program need to be AMODE(24)? What does it do?
Back to top
View user's profile Send private message
Bill O'Boyle

CICS Moderator


Joined: 14 Jan 2008
Posts: 2502
Location: Atlanta, Georgia, USA

PostPosted: Tue Apr 01, 2014 5:45 pm    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

Yes, I second Bill's question.

Is there some reason the called sub-program needs to be AMODE 24?

Is it Assembler with DCB's or does it Call the older-than-dirt COBOL Abend-Program "ILBOABN0"? icon_eek.gif

What version of COBOL are you using?

Please advise....
Back to top
View user's profile Send private message
rajendra kalepu

New User


Joined: 29 Jul 2011
Posts: 21
Location: India

PostPosted: Tue Apr 01, 2014 6:51 pm    Post subject:
Reply with quote

Hi All,

FYI

The program being called contains the retsart logic for all the programs of different applications which are under my account

So changing the sub program to AMODE(31) may affect other applications

Is there any way to pass data form AMODE(31) program to AMODE(24) program

Let me know for more information

Thanks in advance
Rajendra
Back to top
View user's profile Send private message
Bill O'Boyle

CICS Moderator


Joined: 14 Jan 2008
Posts: 2502
Location: Atlanta, Georgia, USA

PostPosted: Tue Apr 01, 2014 6:58 pm    Post subject: Reply to: Need Suggestion in handling huge record limit
Reply with quote

rajendra kalepu asked -

Quote:
Is there any way to pass data form AMODE(31) program to AMODE(24) program

Please review my previous reply regarding a LINKAGE ASSISTANT sub-program.

HTH....
Back to top
View user's profile Send private message
rajendra kalepu

New User


Joined: 29 Jul 2011
Posts: 21
Location: India

PostPosted: Wed Apr 02, 2014 1:12 pm    Post subject:
Reply with quote

Hi Bill,

I am sorry for asking, but as i am new to AMODE(24) and AMODE(31) concepts. Could you please provide me a sample code where we pass data from program compiled at AMODE(31) to program compiled at AMODE(24)

So that i can take it as reference and do the coding accordingly.

I searched for LINKAGE ASSISTANT sub-program for AMODE(31)/AMODE(24)
but i was unable to understand it clearly

Please help me in this regard.

Thanks in advance
Rajendra kalepu
Back to top
View user's profile Send private message
Rohit Umarjikar

Senior Member


Joined: 21 Sep 2010
Posts: 1609
Location: NY,USA

PostPosted: Wed Apr 02, 2014 1:31 pm    Post subject:
Reply with quote

Did you look at the suggestion given by me for a long term fix ?
Back to top
View user's profile Send private message
rajendra kalepu

New User


Joined: 29 Jul 2011
Posts: 21
Location: India

PostPosted: Wed Apr 02, 2014 2:21 pm    Post subject:
Reply with quote

Hi Rohit,

Thanks for your suggestion but problem is all the 3 types must be in Sync i.e Type0 ,Type1, Type 2

So even though i split the file for example if Type 0, Type1, Type 2 for a particular key combination has more than 5000 records the process will fail

Note: I cannot split the Combination for same key into 2

Thanks in advance
Rajendra
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> COBOL Programming All times are GMT + 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Limit duplicate records in the SORT pshongal SYNCSORT 6 Mon Nov 21, 2016 12:54 pm
No new posts How to split the records using the am... vnktrrd DFSORT/ICETOOL 24 Fri Oct 28, 2016 7:33 pm
No new posts Sort records based on numeric field. Alks SYNCSORT 2 Wed Oct 19, 2016 10:14 pm
No new posts abend sort based on count records in ... anatol DFSORT/ICETOOL 5 Mon Oct 17, 2016 10:10 pm
No new posts how to split records based on specifi... Venkata Ramayya DFSORT/ICETOOL 6 Wed Sep 28, 2016 3:20 am


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us