IBM MAINFRAME HELP & SUPPORT FORUMS

dbzTHEdinosauer · Joined: 20 Oct 2006 Posts: 1639 Location: germany

billiions of records is considered needed info when talking about rexx. rexx is not a high end i/o utility. you could compile your rexx.

you could build sort control statements
or
write a cobol/pl1/assembler module to process this data.

vidyaa · Joined: 02 May 2008 Posts: 37 Location: chennai

this rexx consists of panels and rexx codes, where the user input some values i need tovalidate each fields and throw message after validation take the values entered by the user and search if the combination of values entered by the user is present in a file which is going to have billion of records if the combination exists writes it to a file if not pops up an error message as 'Combination not found' and all these will be done online REXX validtions will be the backend so its not possible to put this in a batch mode.Searching of the combinations of field values entered by the user in the file containing billion of records takes me time.

gcicchet · Joined: 28 Jul 2006 Posts: 663

Hi,

I can't believe anyone could come up with such a design, how often are these requests run ? how many users can request this ? what is the current time it takes to satisfy one of these requests ?

Surely the values entered by the user can be validated and saved into a file which can then be used in batch.

I would try to capture all requests into one file and run a batch job on a daily basis.

Gerry

expat · Posted: Tue Aug 05, 2008 12:33 pm Post subject:

Gerry, stop it will you, coming up with sensible suggestions

What do you think this is, a help forum or something :D

dick scherrer · Joined: 23 Nov 2006 Posts: 8732 Location: 221 B Baker St

Hello,

If the concept has been proven, it would be best to do as DBZ suggests and re-code the process in a better suited language.

REXX may be the worst choice for processing high volumes of data. . .

Pedro · Joined: 01 Sep 2006 Posts: 536 Location: work

Quote: search if the combination of values entered by the user is present in a file which is going to have billion of records if the combination exists

If this is searched regularly, I think the data is stored in the wrong format. There is not really a good way to search billions of records stored in a sequential format. It should be in a database, where it might take the reading of a few index blocks before finding (or not) the record.

Craq Giegerich · Joined: 19 May 2007 Posts: 1001 Location: Virginia, USA

Billions of records, Millions of records I think some of these record estimates are a little off.

Pedro · Joined: 01 Sep 2006 Posts: 536 Location: work

Quote: my input has billons od records its very slow on doing this is there any way to make it better and faster.
It is hard to believe that you got to billions of records without first realizing that there were performance problems. When you say 'billions', I am just assuming that you really meant "large number of records".

The way to speed it up is to not use a sequential file. Use VSAM with keyed records. I think you can get tools to process VSAM from rexx. (or as I said earlier, use a real database).

vidyaa · Joined: 02 May 2008 Posts: 37 Location: chennai

yes i got your words intially it was thought to be few record and now it has come to billions. This is need to process online to get the user input and validate and throw him error messages. this cannot be changed to a program i guess. the seraching part kills time is there any way to incorporate this searching part in languale like cobol and combine with rexx as this needs to be done online.

gcicchet · Joined: 28 Jul 2006 Posts: 663

Hi,

I want to be the user entering the data and waiting for the results. :wink:

I'll have to find ways of amusing myself.

Gerry

dick scherrer · Joined: 23 Nov 2006 Posts: 8732 Location: 221 B Baker St

Hello,

Quote: as this needs to be done online Sorry, but i very much disagree.

I believe what is needed is a smooth process for the user to get the needed answers in a timely fashion. That does not make the use of online resources a requirement.

expat · Posted: Wed Aug 06, 2008 11:56 am Post subject:

If it must be done in a wasteful and inappropriate manner then you will need to look at some of the ways to maybe waste a little less resource.

I guess that your shop does not bill each department on its usage of resource, because if they do there will probably be some very unhappy bean counters when the resource bills start rolling in.

If you can use a unique key, then using a VSAM KSDS accessed by IDCAMS within REXX should work.

vidyaa · Joined: 02 May 2008 Posts: 37 Location: chennai

What i meant to say is the user only sees the front screen designed by panels he doesnot know what is happening in the backend rexx he corrects the input if he gets the error message, all the validations are done in rexx and throws the message

expat · Posted: Wed Aug 06, 2008 1:51 pm Post subject:

What happens behind the panels are usually of no interest to the user, just the results.

So how you design / implement a package that delivers the results is of no consequence to the user, which gives you a free hand to search the best and most effective methods.

Pedro · Joined: 01 Sep 2006 Posts: 536 Location: work

Quote: is there any way to incorporate this searching part in languale like cobol and combine with rexx as this needs to be done online.

Yes, you can write the searching part in a compiled language. But a billion records is still a billion records! With a sequential file, I do not think you will ever have a satisfactory process for an online environment. It is the reading of the file that takes a long time and that does not change much regardless of rexx or compiled language. I stand by my recommendation to use VSAM or a database so that less I/O is performed.

A REXX program can call a compiled program to do part of the work. The compiled program can use ISPF VGET / VPUT services to pass information to REXX.