Buffer limit reached while handling huge records

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

1. We receive a file which contains 3 types of records namely Type 0, Type A and Type B
2. All Type0, TypeA, Type contains a common key.
3. We are loading these three types in 3 individual arrays each having a limit of handling 5000 entries (i.e., occurs 5000)
Please find below example for better understanding

input file
0 AAA --> Type 0 Here AAA is the Key for below 3
1 AAA xyz abc --> Type 1
2 AAA def ghi --> Type 2
0 BBB BBB is the key for below 3
1 BBB ghi jkl
2 BBB jkl mno

So once we load Type0 , Type 1 and Type 2 records arrays for a particular Key 'AAA' , we will do some processing and Reset the internal table (assume emtying it)
We will do this for all keys until End of File is reached

Problem i am facing : Currently due to business expansion i am receiving close to 20k records per Key. I mean to say
0 AAA
1 AAA .....
1 AAA .....
:
:
:
10k assume
2 AAA .....
2 AAA .....
:
:
:
10k assume
Total 20k

2. We have an counters for each type (Type0, Type 1 and Type 2) which increments by one on loading the internal tables

3. Due to the exceeding of internal array size i am getting SOC7 at Counter varaible (On analysis i came to know that its because of exceeding array limit)

4. I even tried to increase the limit as a temporary fix but the temporary space allocated for the job is limited (getting an error " Buffer limit reached")
More over i just dont want to increase array limit because i feel its not a good coding practise

Please suggest me a possible solution for this scenario to handle any amount of records in coming future wiht out depending on array limit

It will be very helpful in giving a permanent fix to this issue

Thanks in advance.

Nic Clouston · Posted: Mon Mar 31, 2014 7:41 pm

do you really need all the records for a key in storage at the same time? I.e. are you referring to records more than once? If so ,can you store only the firelds that you need? If not, then you do not need to store them simply read/process/read next

Terry Heinze · Posted: Mon Mar 31, 2014 8:08 pm

As Nic says, try to store only the needed fields of the records you're storing. The compiler limits are very large these days. See Appendix B in the COBOL Language Reference Manual.

dick scherrer · Posted: Mon Mar 31, 2014 8:23 pm

Hello,

Bill Woodger · Posted: Mon Mar 31, 2014 9:00 pm

You need to be explicit about what "Buffer limit reached" means.

If a table needs to hold 20,000 entries, it needs 20,000 entries. In fact, it is good practice to have a growth projection and ensure you don't have to re-visit the same thing in the short/medium term.

You have 128MB of WORKING-STORAGE. How big are your records?

Enterprise COBOL V5.1 has "unbounded" tables (limit not imposed at compile time) but as others have indicated, do you need to hold all the storage anyway?

Bill O'Boyle · Posted: Mon Mar 31, 2014 10:16 pm

Bill,

We had Tom Ross of IBM here doing a COBOL 5.1 overview, so I asked him whether "Unbound" could take on the PICTURE clause maximum of the DEPENDING ON (which in my example, was an unsigned binary-fullword) and he said YES.

I never posted my VSAMXREF sub-program, which you CALL from COBOL and it returns the number of records on the target VSAM file, using a SHOWCB Macro.

HTH....

Rohit Umarjikar · Posted: Tue Apr 01, 2014 12:43 am

You might wants to a multithreading here, split the input file depending upon the "type" , by this you would only have a single "Type" in each file where the records are now controlled and then you can process them parallely using scheduling and at last concatenate all output file into a single file.

Bill Woodger · Posted: Tue Apr 01, 2014 2:24 am

Bill,

I guess that makes sense, although it would be nice if the documentation of UNBOUNDED in the Language Reference was specific :-)

For the OCCURS DEPENDING ON:

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

Hi All,

Thank you for your response

I will try all the possible solutions posted here and will post my results shortly

Thank and Regards
Rajendra

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

Hi All,

Currently we are using "OCCURS DEPENDING" clause only in the program for all the 3 tables.
Actually when i tried to increase the array limit to more than 10000 per table
i am receving the following error during compile time

'THE SIZE OF THE "WORKING-STORAGE SECTION" EXCEEDED THE LIMIT OF 16,777,215 BYTES ALLOWED UNDER THE "RENT" AND"DATA(24)" COMPILER OPTIONS. THE "DATA(31)" OPTION WAS ASSUMED'

The program was existing from past 15 years. We are facing this problem recently because of the business expansion

Please let me know for more information. Do the needful

Thanks and regards,
Rajendra

Bill Woodger · Posted: Tue Apr 01, 2014 1:38 pm

Please include the entire error message, including the error reference which is at the start of the message.

Do you need DATA(24) for your program? With DATA(31) you will have 128MB of WORKING-STORAGE, enough for 20,000+ entries for each table.

If you need DATA(24), explain why.

If you need DATA(24) you'll have to look at the suggestions for cutting things down to size.

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

Hi Mistermind,

Thats great. It worked fine. I coded DATA(31) in my program and i could see the compilation went fine even though the array limit has been set to 20000 records per table.

But i faced problem during the run time. I got an error
"An attempt was made to pass a parameter address above 16megabytes to AMODE(24) sub program"

Unfortunately we are calling a Subprogram defined at AMODE(24) which is used by different applications in our area

Please help me in this regard

Thanks and Regards,
Rajendra

Bill O'Boyle · Posted: Tue Apr 01, 2014 4:24 pm

When a 31-Bit program calls a 24-Bit program, what you could do is write a LINKAGE ASSISTANT program, which is AMODE 31 but builds a 24-Bit Parmlist to be passed to the 24-Bit sub-program, basically as a Pass-Through.

This can be done easily in Assembler, using a GETMAIN or STORAGE OBTAIN Macro, with LOC=24 within the LINKAGE ASSISTANT program and using this Below-the-Line Storage as the 24-Bit Parmlist.

You may also want to look at LE Callable Service routine CEEGTST, but I'm not sure you can specify a particular AMODE in the LINKAGE ASSISTANT, because I believe CEEGTST takes on the characteristics of the Caller.

But, it's woth looking into.

HTH....

Bill Woodger · Posted: Tue Apr 01, 2014 5:30 pm

Does the sub-program need to be AMODE(24)? What does it do?

Bill O'Boyle · Posted: Tue Apr 01, 2014 5:45 pm

Yes, I second Bill's question.

Is there some reason the called sub-program needs to be AMODE 24?

Is it Assembler with DCB's or does it Call the older-than-dirt COBOL Abend-Program "ILBOABN0"?

What version of COBOL are you using?

Please advise....

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

Hi All,

FYI

The program being called contains the retsart logic for all the programs of different applications which are under my account

So changing the sub program to AMODE(31) may affect other applications

Is there any way to pass data form AMODE(31) program to AMODE(24) program

Let me know for more information

Thanks in advance
Rajendra

Bill O'Boyle · Posted: Tue Apr 01, 2014 6:58 pm

rajendra kalepu asked -

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

Hi Bill,

I am sorry for asking, but as i am new to AMODE(24) and AMODE(31) concepts. Could you please provide me a sample code where we pass data from program compiled at AMODE(31) to program compiled at AMODE(24)

So that i can take it as reference and do the coding accordingly.

I searched for LINKAGE ASSISTANT sub-program for AMODE(31)/AMODE(24)
but i was unable to understand it clearly

Please help me in this regard.

Thanks in advance
Rajendra kalepu

Rohit Umarjikar · Posted: Wed Apr 02, 2014 1:31 pm

Did you look at the suggestion given by me for a long term fix ?

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

Hi Rohit,

Thanks for your suggestion but problem is all the 3 types must be in Sync i.e Type0 ,Type1, Type 2

So even though i split the file for example if Type 0, Type1, Type 2 for a particular key combination has more than 5000 records the process will fail

Note: I cannot split the Combination for same key into 2

Thanks in advance
Rajendra

Anuj Dhawan · Posted: Wed Apr 02, 2014 2:30 pm

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

Hi Anuj,

Thank you for the valuable info.

Here the problem is the same subprogram is used by many applications

If i try to compile the subprogram with AMODE(31) . I may not sure may be other application programs which are compiled at AMODE(24) calling this sub program will fail. Correct me if i am wrong

Let me know for more information

Thanks and regards,
Rajendra kalepu

Anuj Dhawan · Posted: Wed Apr 02, 2014 3:14 pm

rajendra kalepu · New User Joined: 29 Jul 2011 Posts: 21 Location: India

Hi Anuj

1. IBM Enterprise COBOL for z/OS 4.2.0 is the version of COBOL we are using

2. Yes we cant change the AMODE for sub programs due to certain restrictions from the account

Please let me know for more information, Thanks in advance

Regards,
Rajendra

Bill O'Boyle · Posted: Wed Apr 02, 2014 4:49 pm

Please give us some idea of these restrictions. As Anuj has said, this is 2014 and to have an AMODE(24) load-module is archaic at best, especially running with Enterprise COBOL 4.2.

I think what you'll find is the old "Well, we always ran it this way", which is a holdover from the OS/VS COBOL days (pre-COBOL2).

It would be best to tackle this issue now, because you'll find additional AMODE(24) restrictions in Enterprise COBOL 5.1, which is now available.

Don't get caught with your pants down.

BTW, this AMODE(24) sub-program is called by other programs? Are these callers also AMODE(24)?