Need Options to Read a file for each record of another file

Kranthi M · New User Joined: 07 Oct 2011 Posts: 20 Location: India

Hi,

I got a Requirement for Performance tuning a program. Issue is that the job is taking 14 hours to run.

This program Reads 2 files. Both are sequential files.

Basic logic that needs change is -
Read FILE1 (approximate number of reacords in this file is 10,000)
For each read of this file read another file (again sequential file) to get a match for Policy Number. (approximate number of records in FILE2 is again 10,000).

FILE1 LRECL - 604 with RECFM VB
FILE2 LRECL - 4000 with RECFM FB

Program has a check for Policy along with 4 other variables. Like below
If POL-FILE1 = POL-FILE2 AND
DATE-FILE1 = DATE-FILE2 AND
IND-FILE1 = IND-FILE2 AND
IND2-FILE1 = IND-FILE2 AND
IND3-FILE1 - IND-FILE2

<DO SOMETHING>
END-IF
When it gets a match then it comes out of READ or READ till End of file.
Then CLOSE FILE2 and OPEN FILE2 for another READ of FILE1.

I need options to minimize the time taken to execute this program.
1) I thought by making FILE2 as VSAM file (KSDS), each Read of FILE1 will have 1 read of FILE2
But I am not sure if this way is good as the file LRECL is 4000.

Kindly let me know if there are other ways of doing it.
I need to give options to my client and then they will get back with a chosen option which I need to code and execute.

Thanks,
Kranthi

dbzTHEdinosauer · Posted: Wed Apr 18, 2012 3:51 pm

if you have the money,
i have the time.

Akatsukami · Posted: Wed Apr 18, 2012 4:13 pm

What incompetent moron of a predecessor decided to use an unsorted FILE2, and open and close it 10,000 times?

Bill Woodger · Posted: Wed Apr 18, 2012 4:15 pm

Bill Woodger · Posted: Wed Apr 18, 2012 5:10 pm

This can still work out.

Your organisation, company, has serious problems.

Someone has written code for a task which should run in less than a minute (depending on machine size, etc) and it runs instead for 14 hours.

The Client knows. How, I don't know, but they know.

Presumably one of the better coders (that's you) has been given the task of providing "options" to the client, but you have no grasp of the basic problem.

Therefore, we can intuit that all systems produced by your organisation are equally poorly equipped to do the job.

So, for an agreed fee, dbz and I can receive your specs and programs. When we have stopped giggling at the code, we can turn them around with an expected minimum of 50% performance improvement, and possibly even more than 90%. Those projections are on what we know so far.

For a further fee, we can produce standards and review processes for your development and testing etc.

The savings for your organisation, and improvement in Client satisfaction, marketability of product, etc, will vastly outstrip what we would charge.

One small rat in the ointment. You work for IBM and somehow someone has produced this and the Client has found out. I'm stunned. Still, run it by your boss, PM us if you are interested :-)

Kranthi M · New User Joined: 07 Oct 2011 Posts: 20 Location: India

Hi,

For many reasons, I am not supposed to give you my program. Spec as I said is to reduce the run time. And for sure code is written long back by some one from Client side.

Also, I guess you are not pointing any thing at IBM and IBMers :-).

I believe that IBM and IBMers produce good quality of code.

And I am really not sure if that FEE thing you are talking about is for real ...

Kindly let me know if there are ways to achieve my task.
I was also, thinking about using SORT but for that to use I need to first understand how FILE1 is SORTED and on same line I need to SORT FILE2.

Also, FILE1 is not a simple sequential file. It stores different types of records and say if there are 10 types of recirds each denoted by some code at the start then type 10 has the information that I process.

These 10 types of records are repeated for each policy.

Kindly let me know if you need more information.

enrico-sorichetti · Posted: Wed Apr 18, 2012 5:36 pm

Bill Woodger · Posted: Wed Apr 18, 2012 5:45 pm

Of course I was serious about the fee :-)

Opening and Closing a file 10,000 times, and reading every record on it 10,000 times, will consume considerable resources. On your client's machine it consumes nearly all the run time for the job, as it would do on anyone's machine. I can't believe that anyone has put up with that for years, but there we go.

The "options" are simple. Do it without opening/closing/reading-the-entire-file 10,000 times.

How best to do it, does depend on the specifics of the data.

Unless the files are not "designed" at all, then multiple record-types should not come into it.

Sort and two-file-match, sort and internal table, sort and less-likely-ksds... the specifics are beyond guessing.

But you, or someone where you work, should know all this...

Kranthi M · New User Joined: 07 Oct 2011 Posts: 20 Location: India

Hi,
I read BCG and posting or pasting directly file names or code is against the BCG.

It is not against when we post in general.

Thanks for your help.

I thought I could get some more options of getting the task done and not in any other way.

It would have been better if there is a tag line that IBMers can not use this forum. But I thought we could use. any how thanks.

enrico-sorichetti · Posted: Wed Apr 18, 2012 5:56 pm

Bill Woodger · Posted: Wed Apr 18, 2012 5:59 pm

Indeed, we have very welcome IBMers on here. They provide solutions, not look for them from us to sell to their clients.

Kranthi M · New User Joined: 07 Oct 2011 Posts: 20 Location: India

Thanks Bill. I started looking into the SORT option already.
Will get back if that code works. And I am sure there are alternatives for all the tasks.

Bill Woodger · Posted: Wed Apr 18, 2012 6:14 pm

And what about thanking the Honourable Akatsukami? Without his intervention (cat spilling) I'd still have wanted that fee...

Akatsukami · Posted: Wed Apr 18, 2012 6:29 pm

enrico-sorichetti · Posted: Wed Apr 18, 2012 6:35 pm

Konnichiwa Akatsukami Sama

Kranthi M · New User Joined: 07 Oct 2011 Posts: 20 Location: India

thanks every one. I can feel only sarcasam. in each and every sentence you have posted. And from now on, I would never want to post any question on this forum. If possible I will post answers ... Actually, I do not wan to use this forum for any reason from now on. Have never seen so much of negativity any where else.

And most of u seem to b Ex-ibmers.... think if you have done any good today, by replying to my post. except venting out your frustration on me.

God Bless you!!

enrico-sorichetti · Posted: Wed Apr 18, 2012 7:06 pm

Kranthi M · New User Joined: 07 Oct 2011 Posts: 20 Location: India

I wish I could understand what you are saying. But just because I am IBMer for some time does not mean ...you should say all these.

If I resign in a weeks time from now....then will you answer for my questions in more nice way....

and How does it make any difference to you...me being an IBMer now and in a week out of IBM.

And will it make any difference if I am from IBM India or from some other company.

Yes, I became a victim of all your offences...for no real good reason ....

And again, I have never ever seen this kind of sarcasm and negetivity any where else or from any person for no good reason...

We are all earning because we are working on IBMs product IBM Mainframes. I am thankful to it. I will be even if I join some other company.

Talking about ethics ....we can still go back and read forum rules...and think about the real ethics we are following.

I am sure you still will not accept and will try to reply again. Dont you think, with your level of knowledge you could have even answered to my question in last 2 hours. Instead you have been posting just to make some comments about IBM.

I have all the respect for all the companies that I have worked with and for all the people that I have interacted with including you...considered you to be knowledgeable person....because I saw that you are a moderator for this forum.

Jose Mateo · Posted: Wed Apr 18, 2012 7:40 pm

Good day to all!

I can not believe a program badly design as it is take 14 hours to run with 10,000 records in each file. First, I really believe it was design like that to su** the milk out of the cow (client) but the cow got wise as time went by.

Bill Woodger · Posted: Wed Apr 18, 2012 7:54 pm

Kranthi,

It is not personal, if you read.

What we do tend to dislike is consultants/consultancies coming here for solutions to problems they can't handle themselves, which we know at the end of the day they are going to make money from by passing the solution off as their own and the client paying.

I did not realise initially that it was a client-written program. Still, the problem itself was so obvious as to not require asking here, surely?

If, when you had signed-up here, you had never mentioned IBM as an employer, and if, when asking the question, you had never introduced the "client" part of the relationship, the responses would have been somewhat different, but we'd still have expected it to be obvious to you what the "basic" problem was.

Anyway, whatever happend to that famous old "IBMer Sense of Humour" you used to all be known and loved for? :-)

EDIT: I've just read to the end of your previous post.

The point about the solution is that it is instantly obvious. Don't, and don't ever, do such a silly thing. I know it wasn't you, but you should know. Hopefully you do now.

Beyond that, we can say no more as you are unable to provide anything more. We can't provide more direct answers to an entirely open-ended question.

dick scherrer · Posted: Wed Apr 18, 2012 11:23 pm

Hello,