Mainframe flat file splitting if maximum size limit reached

yogeshwar_ade · Active User Joined: 31 Aug 2006 Posts: 103 Location: INDIA

I have a file say 'MF.ABC'. If it reaches size of say 500 MB then we need to split this file into two files to avoid SIZE issue during FTP.

We are trying to FTP MF.ABC file but maximum allowable size is 500 MB, if it exceeds then FTP get failed. Due to which we need to split this file if above SIZE limit reaches.

Please let me know if any other details required.

Thanks,
Yogeshwar (Yogi)

superk · Posted: Wed Jan 23, 2013 8:55 pm

I don't personally know of anyway to directly check if a file exceeds a particular number of bytes. I'd suggest checking for a maximum record count.

Bill Woodger · Posted: Wed Jan 23, 2013 9:11 pm

Which SORT product do you use? What is the RECFM and LRECL? What processing do you already have of the file so that something new doesn't cost "extra" resources?

yogeshwar_ade · Active User Joined: 31 Aug 2006 Posts: 103 Location: INDIA

Bill,

We are using SYNCSORT product. RECFM is FB and LRECL is 1000 for this file.

Bill Woodger · Posted: Thu Jan 24, 2013 6:52 pm

So, consult your documentation for SPLIT, SPLIT1R and SPLITBY (if they all exist for Syncsort).

500mb of 1000-byte records is a certain number. You have various options to arrange for multiple files which could be transferred separately.

I'd arrange for headers (logical name, logical number, business/data date) and trailers (count on file) at a minimum, so that the receiver knows that files are logically the same data. Stuff like that.

yogeshwar_ade · Active User Joined: 31 Aug 2006 Posts: 103 Location: INDIA

You mean to say get record count for 500 MB (since its 1000 byte LRECL) and once we got record count go ahead with SORT using SPLIT option to split, right?

Bill Woodger · Posted: Thu Jan 24, 2013 7:59 pm

Yep. You have a limit in MB for the transfer (site specific). It is "easy" to calculate a number of records which would keep you within the limit. If there are "groups" of records which need to be kept together, that is not a problem as we've already answered a couple of those recently.

So, yes, but I'm also saying do it properly, with "controls" around it, so that if there is a mess-up on the receiving system, you are covered because you've provided controls over the data.

yogeshwar_ade · Active User Joined: 31 Aug 2006 Posts: 103 Location: INDIA

My apology for delay in reply.

This file is not having any Header and Trailer and also we didn't need to group any specific records.

I guess now I need to first find how many records in 500 MB for 1000 LRECL file of FB type. If any pointers over how to calculate record in 500 MB will be helpful.

Secondly, I need to split file based on number of records. Say if there are 1 Million record for 500 MB file size. So for very 1 million record there will be new file to be created. If we are getting total 9 million record in file, then this file need to be splitted into 9 difdifferent files having 1 million records each.
The records in this file may vary and some time come to say 9 million and some time 20 million and we need to split this file accordingly, is there anyway we can achieve this one whether thru JCL/SORT or COBOL?

Please let me know if I missed something and requirement is not clear.

Robert Sample · Posted: Thu Feb 07, 2013 6:44 pm

If you do not know how to calculate the number of records, then you belong in Beginner's and Students Forum no matter how much "experience" you have.

For a disk file on a 3390 device, half-track blocking means 27 1,000-byte records will fit in each block. This means 54 records per track and 810 records per cylinder. If you can figure out how many 1,000-byte records will occur in 500 MB, you can finish the computations yourself.

If using tape, the numbers are different but the calculation is the same (although, obviously, tracks and cylinders are not part of the calculation).

yogeshwar_ade · Active User Joined: 31 Aug 2006 Posts: 103 Location: INDIA

My miss Rob, I missed the basic rule and posted the query in hurry.

Bill Woodger · Posted: Thu Feb 07, 2013 7:28 pm

500mb = 500,000,000 (approx). LRECL 1000. Equals 500,000 records.

You specify multiple OUTFIL datasets (at least 40 is you might have 20 million records, but I'd allow for "growth" as well, if there can be any).

You'd need to know how many files were output, remove the rest, and process those with data.