Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

IGYDS1267-S, IGYGR1478-E
Goto page 1, 2, 3  Next
 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> COBOL Programming
View previous topic :: :: View next topic  
Author Message
kunal jain

New User


Joined: 19 May 2011
Posts: 59
Location: India

PostPosted: Thu May 19, 2011 1:16 pm    Post subject: IGYDS1267-S, IGYGR1478-E
Reply with quote

Hi,

I am getting the below two compilation error.
IGYDS1267-S THE SIZE OF THE "WORKING-STORAGE SECTION" EXCEEDED THE COMPILER LIMIT OF 128 MEGABYTES OF MEMORY. EXECUTION RESULTS ARE UNPREDICTABLE.

IGYGR1478-E THE SIZE OF THE "WORKING-STORAGE SECTION" EXCEEDED THE LIMIT OF 16,777,215 BYTES ALLOWED UNDER THE "RENT" AND "DATA(24)" COMPILER OPTIONS. THE "DATA(31)" OPTION WAS ASSUMED.

I know that my working-storage size is going beyond limit because I am using two arrays(working-storage table) of 50000 records each and each record contains 165 chars.
It is almost impossible for me to get rid of this 2 tables.
Please help me in this by suggesting any way to increase our work-storage area and to remove the above 2 errors.
Back to top
View user's profile Send private message

enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10202
Location: italy

PostPosted: Thu May 19, 2011 1:41 pm    Post subject: Reply to: IGYDS1267-S, IGYGR1478-E
Reply with quote

Quote:
Please help me in this by suggesting any way to increase our work-storage area and to remove the above 2 errors.


as described by appendix b of the cobol reference manual here
http://publibz.boulder.ibm.com/cgibin/bookmgr_OS390/BOOKS/IGY3LR31/CCONTENTS?SHELF=IGY3SH33&DN=SC27-1408-04&DT=20060329140556
compiler limits are that ... LIMITS and cannot be changed


rereading the topic
what happened when in both cases You/Your support forced DATA(31)

after all unless my arithmetic is completely FUBARed
50000 times 165 gives 8250000 which is a bit less than 8388608 which are 8MB
and two of them should fit nicely in 128Mbytes (134217728 ) less a few control bytes

unless You made a typo and a zero is missing somewhere ?

so while it is ok to fail in the second case, it should not in the first, look around for other symptoms

if You made a typo then sadly all You can do is review the program design
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Thu May 19, 2011 2:25 pm    Post subject:
Reply with quote

welllllll,
let us assume you have 1 module,
and in this module you want two (2) gigantic cobol internal tables.

but it will not all fit into one module.

so,
a solution could be

primary module:
Code:

WORKING-STORAGE  SECTION.

...

01  GROUP-AREA-POINTERS.
    05  POINTER-TO-TABLE-A      USAGE IS POINTER.
    05  POINTER-TO-TABLE-B      USAGE IS POINTER.
...

LINKAGE         SECTION.

01  GROUP-AREA-DEFINING-TABLE-A.
    05  TABLE-A-ITEM            OCCURS n TIMES
                                INDEXED BY TABLE-A-INDEX.
        10  TABLE-A-FIELD1      PIC ?
        ...

01  GROUP-AREA-DEFINING-TABLE-B.
    05  TABLE-B-ITEM            OCCURS m TIMES
                                INDEXED BY TABLE-B-INDEX.
        10  TABLE-B-FIELD1      PIC ?
        ...

PROCEDURE DIVISION.

...

CALL PROGRAM-A
     USING POINTER-TO-TABLE-A
END-CALL
CALL PROGRAM-B
     USING POINTER-TO-TABLE-B
END-CALL

SET ADDRESS OF GROUP-AREA-DEFINING-TABLE-A  TO POINTER-TO-TABLE-A
SET ADDRESS OF GROUP-AREA-DEFINING-TABLE-B  TO POINTER-TO-TABLE-B

...



PROGRAM-A (you can clone PROGRAM-B from this definition)
Code:

WORKING-STORAGE       SECTION.

01  GIGANTIC-TABLE.
(here use the same table definition as used by primary program LINKAGE SECTION
item GROUP-AREA-DEFINING-TABLE-A)
...

LINKAGE    SECTION.
01  POINTER-FROM-CALLING-PROGRAM     USAGE IS POINTER.

PROCEDURE DIVISION USING POINTER-FROM-CALLING-PROGRAM.
SET POINTER-FROM-CALLING-PROGRAM TO ADDRESS OF GIGANTIC-TABLE
GOBACK.



essentially, all you are doing is accessing program-a's and program-b's working storage from primary module.
Back to top
View user's profile Send private message
kunal jain

New User


Joined: 19 May 2011
Posts: 59
Location: India

PostPosted: Thu May 19, 2011 2:45 pm    Post subject:
Reply with quote

ENrico,
Yes, It was a typo. There are 2 arrays(working-storage table) of 500000 records each and each record contains 165 chars.



dbzTHEdinosauer,
Thanks for your reply..but i would appreciate if you can elaborate more on it.
What i understood is you want me to use 3 modules-primary,A & B.
In that case why primary requires Linkage Section.

Please explain this in more easy way.

Thanks in advance!
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 7913
Location: Bellevue, IA

PostPosted: Thu May 19, 2011 2:59 pm    Post subject:
Reply with quote

Quote:
It is almost impossible for me to get rid of this 2 tables.
Please help me in this by suggesting any way to increase our work-storage area and to remove the above 2 errors.
Dick is trying to find a way to break up the tables so they are in separate programs and hence will fit into the 128M limit. Since you CANNOT increase the WORKING-STORAGE limit, you have to either (1) reduce the size of the tables, or (2) use more than one program to keep each program's WORKING-STORAGE under the limit.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7233

PostPosted: Thu May 19, 2011 3:02 pm    Post subject: Reply to: IGYDS1267-S, IGYGR1478-E
Reply with quote

I hope Dick doesn't mind be butting in.

He has given you a very clear, maintainable solution to your problem, and one which once you know it also has other applications.

In his description, he is already clear through the use of well-chosen names and reasonable comments.

Your program contains two very large tables. It would allow your program to compile if the tables were not in your program, but somewhere else, where you could use them.

So, you need somewhere else to put your tables.

Pre-ADDRESS OF, the way to do this would be for your program to be a sub-program. Program A calls B with first Gigantic Table. Program B calls your program with Gigantic Table from its LINKAGE (from Program A) and its own Gigantic Table from WORKING-STORAGE. Your program has two Gigantic Tables in LINKAGE and names them on the PROCEDURE DIVISION USING.

So your program uses storage defined in A and B and gets along happily.

With ADDRESS OF, you can achieve the same effect, with more flexibility, by calling A and B, which will each then give you the ADDRESS of the Gigantic Tables that they define, which is all you need to be able to use that storage in your program. Look at is as having access to three Working Storage's. Or more, if you need.

You do have a further possible solution using LE Heaps to get the storage.
Search in this forum for that solution.

For a use like yours, I think Dick's solution is the one.
Back to top
View user's profile Send private message
kunal jain

New User


Joined: 19 May 2011
Posts: 59
Location: India

PostPosted: Thu May 19, 2011 3:12 pm    Post subject:
Reply with quote

Bill Woodger,

It seems that single table of 500000 records also not able to suffice the storage.it is giving the same error.

Could you please explain me the use of LE Heap to solve this issue.

It would be really helpful!''Thanks
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7233

PostPosted: Thu May 19, 2011 3:21 pm    Post subject:
Reply with quote

kunal jain wrote:
Bill Woodger,

It seems that single table of 500000 records also not able to suffice the storage.it is giving the same error.

Could you please explain me the use of LE Heap to solve this issue.

It would be really helpful!''Thanks


Firstly, 500000 * 165 = 82.5 meg. Less than 128mb. Are you using DATA(31) compile option to get rid of the 2nd message?

As for the HEAP option, I suggested searching this Cobol forum. Box towards top right of the screen. Put in HEAP I guess.

What are you putting in the tables?
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Thu May 19, 2011 3:48 pm    Post subject:
Reply with quote

over the last 2 or 3 decades, I have written a few programs on 3 different types of mainframe hardware. only once ran out of space, after which i rehoned my progamming techniques.

what will happen when you need to store more than 500k ITEMS - records are things found in files, not db2 tables, not cobol internal tables, ONLY FILES (and table spaces).


500k items is a lot.
165 bytes for each item in a cobol internal table, is a lot.

If you know that you will always need an enormous amount of memory for your program to process,
it is better to acquire it thru working-storage.
If sometimes you need a lot, and sometimes you don't,
using the LE getmain /get memory functions are better.

I can only assume that you have rudimentary skills, since you did not comprehend my suggested solution.

based on that (prejudice),
I can only assume that you do not have a good design concept.

if you explain what your requirement is,
we could help with the design.

there are other methods that you can employ:
  • break the items up into 2 or more tables - each table at occurance x will comprise the complete item in question
  • use file match merge processing
  • ...incomplete list of suggestions due to lack of information of the requirement


one of the problems of large memory acquisition is paging.
after a while, paging can slow your program down (and the system as a whole)
and you will have to redesign, anyway.

like most posts, i imagine you will ignore the request for more info,
because you want to do it your way.
if so, you have been given solutions to your space problem.

also, PGM-a and PGM-b only needs one working-storage entry,
01 table-area pic x(128,000,000). (or 127,999,990 - whatever)
no need to define the table with occurs structure in Pgm-a and pgm-b.
Back to top
View user's profile Send private message
kunal jain

New User


Joined: 19 May 2011
Posts: 59
Location: India

PostPosted: Thu May 19, 2011 4:00 pm    Post subject:
Reply with quote

Basically I need to compare the records of two VSAM files.
This comparision is a bit complex in a sense that
1. there are some acceptable differences which i need to ignore
2. Record has a. item key b. item description. one item key may contain lacs of description distinguished by sequence number..
3. the description coming in two files for an item key is not is sequence. So to compare all the description for an item-key, I am planning to store them in two internal tables. then i am planning to check for an item-key if every description of 1st Table is present in 2nd table(anywhere) or not.If not present in 2nd Table then it will go to output report as Deleted description and for vice versa case as Inserted record.

Hope this helps you to give an idea of my requirement to help me.

Please suggest

thanks
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Thu May 19, 2011 4:28 pm    Post subject:
Reply with quote

in that case i would consider the following:

growth of files means that eventually you will encounter more items than you can store.

would determine what data i need at any time to make a decision,
if necessary making up new KEYS.

i would extract all the data from both vsam files via SORT, creating qsam files.

i would reformat where necessary,
making duplicates where necessary,
and append the 'new keys' where appropriate.

sort both files so that my program would involve match merge logic,
storing - on-the-fly - only the data necessary for the current key match processing.
that means potentially, reloading the cobol internal tables for each 'key match'.

i would buffer the qsam files up to a max (BUFFNO=? dcb parm on input files).

then the program would consist of:
  • load data for current key
  • process current key
  • start loop again

in that way, your requirement for a large program (cobol internal table) would be reduced.

a lot of work may be off-loaded to sort, inorder to remove those situations where your program would normally ignore something.
Back to top
View user's profile Send private message
kunal jain

New User


Joined: 19 May 2011
Posts: 59
Location: India

PostPosted: Thu May 19, 2011 5:12 pm    Post subject:
Reply with quote

Thank you for your suggestion !
But really speaking I didnt get the clear idea. If its possible for you to elaborate or to be in detail about what you are trying to say..

Also could you please suugest a solution or a way in my design thinking...
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Thu May 19, 2011 6:07 pm    Post subject:
Reply with quote

Kunal Jain,

i don't have a clear idea of your requirement,
so I suggested a fuzzy solution to what to me was a fuzzy requirement description.

with luck, others may have divined the precise requirement from your descirption,
and can help you.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7233

PostPosted: Thu May 19, 2011 6:31 pm    Post subject: Reply to: IGYDS1267-S, IGYGR1478-E
Reply with quote

As Dick as suggested, the first thing to do is to break it down.

With a complicated program, this is always what I do. What I mean here by "complicated" is doing lots of different things. If you do that in one program, you will get more errors, as the more you code, the more errors you get. Plus, when you run, any part of it fails, the whole lot fails. Plus performance. Lots of things to help make things simple, maintainable and convenient.

There is nothing in the world stopping you copying your VSAM files to QSAM files (flat).

If one or both need a new key, give it one. If one or both then need to be resequenced for your requirment, stick 'em through a sort. If you need to "do something" that is only possible by resequencing, yet you need it in your program in the original order, then add a sequence number, sort on the other key and do your processing, then sort on the sequence number to get back to the original order.

With simple extract/sort, get the files to the ideal state for your program. You are not (or should not be) constrained by the way the files are currently designed/held. You can get the files how you want them. Do it in simple steps. Easy to understand. Easy to test. Easy for re-run. Maintainable.

Once you have files ready and reformatted for your program, read doing the match as you go along. Do the match, which is the important part of the logic, which you use the format/sequence of the data to aid your requirement.

Write your output as flat file(s) as well.

Finally, stick 'em (if required) into nicely defined VSAM file(s) (ie defined logically from how the file(s) will be used).

Done.

We can only give this sort of outline, which is what Dick has already done, without having the full spec (which we don't want unless you get really stuck).

Again:

In as many simple steps as are needed, extract/format/sequence the files/data you require for your processing.

Do your matching program.

Produce a fine VSAM file optimised for how it will be used (if necessary).

Keep everything simple. If something gets complicated, simplify it. Work out your full design, starting from rough outline (as already provided in the forum) and firm it up until you can go through the design and know that a "system" following your design will work.

Only then pick up your coding pad and pencil... icon_smile.gif
Back to top
View user's profile Send private message
kunal jain

New User


Joined: 19 May 2011
Posts: 59
Location: India

PostPosted: Thu May 19, 2011 6:51 pm    Post subject:
Reply with quote

Thank you Bill,

I will reframe my design ae per above approcah and will progress with it.

Thanks All... icon_smile.gif for answering all my doubts..
Back to top
View user's profile Send private message
kunal jain

New User


Joined: 19 May 2011
Posts: 59
Location: India

PostPosted: Mon May 23, 2011 6:22 pm    Post subject:
Reply with quote

Hi Dick Brenholtz ,

I have tried using your approach of 3 modules as per you post above at Posted: Thu May 19, 2011 2:25 pm

Now it is showing the same compile error for Linkage Section of Primary modules since the linkage section of primary module contains 2 gigantic table declaration.
IGYDS1267-S THE SIZE OF THE "LINKAGE SECTION" EXCEEDED THE COMPILER LIMIT OF 128 MEGABYTES OF MEMORY. EXECUTION RESULTS ARE UNPREDICTABLE.

Please look into this and let me know your valuable suggestion.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7233

PostPosted: Mon May 23, 2011 6:41 pm    Post subject:
Reply with quote

You have two tables, each occurs 500,000, 165 bytes per table element (from memory).

That is how you need to define the tables that allocate the storage.

The tables you have in the Linkage Section do not allocate storage.

But your tables exceed 128mb - that means that the definition only exceeds 128mb.

Now, the problem is that the Cobol compiler allocates address areas to refer to Linkage (BLLs, which I always understood to be called Base Locator Linkage, but don't quote me). I guess the compiler is warning you that it might not be able to reference everything defined in your linkage section through its BLL scheme.

However, this need not matter.

If you really need to go this route, here's the trick.
Code:

01  GIGANTIC-TABLE-1.
      05  GT1-TABLE-ENTRIES OCCURS 1 TIMES.     
           10  GT1-ENTRY PIC X(165).
01  GIGANTIC-TABLE-2.
      05  GT2-TABLE-ENTRIES OCCURS 1 TIMES.     
           10  GT2-ENTRY PIC X(165).


Mmmm... you say. But I wanted 500,000 of them. Well, make sure your subscript is big enough to hold 500,000. The Linkage Section does not define storage, so there is no need in the Linkage Section to have (in this case) more than one occurs.

Of course, you'll need SSRANGE off and you won't be able to use SEARCH ALL with Occurs Depending On. You can write your own "binary chop" search.

I still think you're better off going down a different route, but if you have to do this, then you have to cut corners, like this one.

If you do this, make sure it is well-documented inside and outside the program.

I wouldn't bother to do the Linkage that way generally, as you are loosing information (for the human reader) about the table. Hence, document it whereas normally you woulndn't need to.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7233

PostPosted: Mon May 23, 2011 7:02 pm    Post subject:
Reply with quote

Whoops, didn't explain the significance of the BLL.

Code:

01  LINKAGE-ITEM-1 PIC X.
01  LINKAGE-ITEM-2
      05  FILLER PIC X(4096).
      05  LI2-ITEM PIC X.
01  LINKAGE-ITEM-3 PIC X.
01  LINKAGE-ITEM-4.
      05  FILLER OCCURS 1000.
            10  LI4-ITEM PIC X(5).
01  LINKAGE-ITEM-5 PIC X.


Each 01 level gets a BLL (look at your DMAP listing). The BLL is used with a "displacement" of three hex digits (000-FFF) which allows each BLL to reference 4096 bytes of storage. If you look at LI2-ITEM it will have a new BLL.

Effectively, the Cobol compiler is using the BLLs to reference individual fields. However, for fields which are part of an Occurs, it only uses the BLL of the starting point, and calculates using that address and the number of the occurrence you are interested in with reference to the current subscript/index you are using. For your huge table, the compiler would generate many BLLs which it would never generate any code to use (I don't know if the Optimizer would clean them up?).

Bear one additional thing in mind. We've been showing you how to get around the storage limits imposed by the compiler. There is no implication that we have ever done this with tables of your size. You might get to a dead-end. If so, don't blame us. You can ask us, maybe we can think of something else, but there is no guarantee that we have done what you are doing with tables of your size.

If you are still, still, going this route, I think the first thing you need to now do is thoroughly test out the addressability in Cobol of your two tables. You don't want to find out that you can't do it after you've coded everything for that specific approach.

Did you change your compile option for 31-bit data?
Back to top
View user's profile Send private message
kunal jain

New User


Joined: 19 May 2011
Posts: 59
Location: India

PostPosted: Mon May 23, 2011 7:08 pm    Post subject:
Reply with quote

Thanks Bill for briefing about the concepts of BLL and all your explanation on the issue.

Yes , I am using the compile option for 31-bitdata

I am trying this approach to see if get my desired results.

Thanks a Ton!!
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Mon May 23, 2011 7:10 pm    Post subject:
Reply with quote

some shops do not allow NOSSRANGE.

though what you have provide, BILL, will work (and I have used it myself, sometimes),
another possible solution
(if you do not have to reference both tables in the same statement)
is use one 01, use two 05's, one redefining the other, and
flip-flop between table a and table b with a set (the 01 linkage item) to pointer phrase.

actually, I agree with Bill, and a non-large table process be used.
problem with two 500,000 165-char item tables,
you are trying to load everything.
as I said before,
what are you going to do whtn 500,000 165-char items is not enough?

since you need to compare both at the same time,
move an item from table a to working storage
flip the address
move an item from table b to working storage,
do the compare.

but again,
you think that you can load the world to memory and play with it.

can't always do that, might as well learn how to design a complete process
instead of attempting to solve the problem in one program.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> COBOL Programming All times are GMT + 6 Hours
Goto page 1, 2, 3  Next
Page 1 of 3

 

Search our Forum:



Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us