PS file storage capacity

depakjan · New User Joined: 14 Apr 2008 Posts: 7 Location: chennai

How many million records can a PS file contain?

I have a requirement of merging 32 files having around 5 million records each... each record will inturn have a length of 3500... is this possible in a single step? .. is the storage capacity of a single PS file mainframe dependent.. i mean will size differ with the configuration of mainframe?

himanshu7 · Posted: Mon Apr 06, 2009 11:09 am

Hi,

depakjan · New User Joined: 14 Apr 2008 Posts: 7 Location: chennai

yea i could have... but i dont have a sample file... thats my prob.. i just wanna know if any of guys have done such merging ...

also which would be a better approach....

i need to merge three group of 32 files and get three files individually... running three merge steps sequentially in a single JCL or running them as three jobs at the same time?

himanshu7 · Posted: Mon Apr 06, 2009 12:01 pm

Anuj Dhawan · Posted: Mon Apr 06, 2009 12:12 pm

Robert Sample · Posted: Mon Apr 06, 2009 5:05 pm

32 files times 5 million records times 3500 bytes per record equals 560 billion bytes.

A sequential file on DASD (up through z/OS 1.9) is limited to 65,535 tracks and 59 volumes. 65535 * [27998/3500] * 2 * 59 volumes yields just under 190 billion bytes (the [] refers to truncation). Your file will not fit on DASD under z/OS unless you're running z/OS 1.10 with Extended Address Volume. Even if you are, putting a 560 billion byte file on disk suggests consulting your site support group to ensure the DASD space is available and how to get the file allocated; a special storage group may need to be defined for the file.

If you're not running z/OS 1.10, you'll have to put the file on tape, or give up the idea of creating a single file.

depakjan · New User Joined: 14 Apr 2008 Posts: 7 Location: chennai

Thanks for the gr8 replies guys... and if i am storing that single file in the tape... i can still do some manipulations using ICETOOL comparison right?.. guessing it would take a hell lot of time..

dick scherrer · Posted: Tue Apr 07, 2009 9:53 am

Hello,

Depending on your exact requirements, you may want to extract the actual needed data from the whole file. I suspect that of 3500 bytes per record, only a few fields are actually needed for this process.

A lazy way to code is pass all of the bytes from all of the records.

Some time spent designing the process to reduce the sheer volume of data to be processed is in order (if not required).

If you post more details of your processing requirement, someone may have suggestons on how to more efficiently implement.

Robert Sample · Posted: Tue Apr 07, 2009 5:28 pm

Be aware there's a limit of 255 volumes to one file, so each tape needs to store at least 2.2 billion bytes to get the 560 billion bytes into 255 volumes. Otherwise, you'll have to extract data (either fields to reduce size of each record, or records to reduce record count, or both) to get the file down to a manageable size.

And processing half a terabyte would definitely take one hell of a long time. I'm seeing FICON transfer rates of 100 MB per second, which is 3.6 GB per hour (actually less due to channel contention and two-way traffic, so let's assume 3 GB per hour). 521 GB divided by 3 GB per hour yields elapsed time to read the file of 174 hours or just over one week. ESCON is 17 MB per second so if you're on an ESCON channel the time becomes 6 weeks to read the file one time.

Terry Heinze · Posted: Tue Apr 07, 2009 10:17 pm

Robert Sample · Posted: Tue Apr 07, 2009 10:39 pm

Back when I worked for a software vendor, one of our customers was getting ready to implement our system into production but backed off when they realized the projected daily batch job run time was 30 hours. That required a bit of redesign to resolve ....

dick scherrer · Posted: Tue Apr 07, 2009 11:29 pm

If it takes more than 24 hours to run the "daily", how long will it take to run the "weekly" . . .?

Heaven help us at year-end. . . .

Robert Sample · Posted: Tue Apr 07, 2009 11:33 pm

We weren't running any weekly jobs, but the month-end job was not fun.