how to increase the efficiency of processing

anand tr · New User Joined: 12 Aug 2008 Posts: 41 Location: chennai

hi,
i have 2 files, one flat and another a KSDS.
In flat file am having around 90k records and has LRECL as 400, amoung which i have a table for stock nos which occurs upto max of 30 times.
now the objective is to create seperate record for each stock no along with other fields.
eg:
input- abcd3101efg102hij103
4th field gives the no of stock no's present which is 3 in this case.
desired output in KSDS-
abcd101efghij
abcd102efghij
abcd103efghij
Wat i have done is reading the flat file, moving the required fields into the key field of KSDS which i have opened in I-O mode,and reading the KSDS based on this key.If valid key then am updating (using rewrite)
else am write the new record.
but with this method even though am getting the desired o/p ,the time taken is around 20-25mins which i feel is huge. could anyone suggest so as to how can i increase the efficiency?

expat · Posted: Wed Oct 15, 2008 5:11 pm

Take a think about it, 90,000 records in 20 minutes is about 4,500 records per minute, which is about 75 records per second.

You read one file extract info from that then perform a read on a second file and check the results, and then determine further processing on the outcome of the check and then perform that process, 75 times a second.

What's so huge about the timescales here ?

Robert Sample · Posted: Wed Oct 15, 2008 5:12 pm

Look at the definition of the VSAM file, look at the CI and CA splits, check the JCL to ensure the vSAM buffering is high enough to support your requirements, run the program through STROBE or another run-time analysis tool, accept that 20-25 minutes isn't necessarily a bad amount of time for the job. With the limited amount of information you've provided, all we can provide are generic answers.

anand tr · New User Joined: 12 Aug 2008 Posts: 41 Location: chennai

Hi expat,
Its true that processing 75 times a second is not a bad efficiency.
But the thing is that during test run am point to a flat file, but the actual requirement is to point to a gdg base which has around 90 generations.
Hence am worried in meeting this requirement.

expat · Posted: Wed Oct 15, 2008 5:51 pm

Taking a second look at this, and considering the points raised by Robert, I would certainly make sure that the input file is sorted efficiently by the KSDS key to ensure that the same CI is not loaded, updated, overwritten in storage and then loaded and updated once again.

Increasing the buffer numbers may help or may extend the elapsed time, as I see that your program reads a record and creates up to 30 KSDS keys. These keys could theoretically be in a different CI from each other, and would need a BUFND of at least 30, which considerably increases the amount of virtual storage demanded by the job, which may then in turn cause paging in/out to occur. It is a fine line between the right number of buffers and too many / too few buffers and the impact that this has.

Maybe it might be better to create all of the KSDS keys in one program, sort them by absolute key value and then perform the updates / inserts. This way you will be certain that no CI will ever be overwritten in storage and then reloaded at a later time.

As suggested a run time analysis tool would certainly give you a lot more additional information.

Robert Sample · Posted: Wed Oct 15, 2008 6:12 pm

expat · Posted: Wed Oct 15, 2008 6:22 pm

anand tr · New User Joined: 12 Aug 2008 Posts: 41 Location: chennai

Yes expat, you are right. For one particular record read from flat file i may get upto 30 keys.
As per your suggestion, my understanding is to read the flat file, generate the keys write it into another flat file, next stepis to sort on keys followed by a 'REPRO' to a KSDS.
Am i right?
I also tried tracing the flow of the prog, using expeditor and the flow was fine as per my knowledge.
So moving forward, can you suggest me a method to determine the CI size?Also could you please suggest, where to include the BUFNO?

[/quote]

anand tr · New User Joined: 12 Aug 2008 Posts: 41 Location: chennai

Hi Robert,

expat · Posted: Wed Oct 15, 2008 6:38 pm

If you do not need the data that currently exists on the KSDS file as input to your file updates, then that should work.

However to cut processing even more, why not sort out directly to the KSDS ?

Robert Sample · Posted: Wed Oct 15, 2008 6:44 pm

IDCAMS LISTCAT command for the VSAM cluster will tell you the CI size as well as splits. In your batch JCL on the VSAM DD statement, include AMP=('BUFND=??,BUFNI=??') to set the data and index component buffers. For the flat file setting BUFNO=30 might speed up processing as well -- at a cost of memory (hopefully your machine isn't memory constrained).

anand tr · New User Joined: 12 Aug 2008 Posts: 41 Location: chennai

The job should run daily and i should have the entire months records in the KSDS. So am specifing DISP=MOD.
So if todays data contains a duplicate key, i guess, i wont be able to write(Update) into KSDS.In that case shud i go for a cobol code again?
Please correct me if i am wrong.

expat · Posted: Wed Oct 15, 2008 6:53 pm

DISP=SHR will suffice for a KSDS irrespective of the operation being performed.

Do you mean if the key already exists and you want to update the record, I believe that you can still replace the record if the same key exists using SORT.

anand tr · New User Joined: 12 Aug 2008 Posts: 41 Location: chennai

Terry Heinze · Posted: Wed Oct 15, 2008 11:26 pm

Agree with everything so far and in addition:
If your "not found" read attempts exceed your "found" attempts by quite a bit, attempt to write a new record first, instead of looking for an existing one. That would save a little I/O, but the other suggestions are better.

anand tr · New User Joined: 12 Aug 2008 Posts: 41 Location: chennai

can anyone suggest me so that i can go ahead writing/updating(in case of duplicate keys) KSDS using SORT?