ICETOOL space issues

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

hello, i am using icetool to split up different record types from input file using different control card.

First two control statements were executed successfully..but third is having issues.

Pasated below sort messages. this also has control cards used in sort

Any suggestion to better code sort card.

rakesha.hg · Posted: Tue Jul 10, 2012 3:00 pm

how about adding some sort work files ??

Gnanas N · Posted: Tue Jul 10, 2012 3:05 pm

Try increasing REGION size for the ICETOOL step/job.

IIRC, it's recommended not to code SORTWORK data sets; let DFSORT handle the space allocations as needed.

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

Work areas 1 to 60 were tried, still no success

so checking if any issue with sort card

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

For AA record

Bill Woodger · Posted: Tue Jul 10, 2012 3:43 pm

Is that all the cards? Are you using ICETOOL like your subject says, or not?

You are using INCLUDE on OUTFIL. That means you are sorting things which you do not need.

Use a plain INCLUDE/OMIT so that records not required are excluded before the sort.

The main difference in the control cards is the huge key one the final one. I'm not really sure how that works with the shorter records - is that why you have VLSHRT?

I think using INCLUDE/OMIT correctly will improve things greatly.

You do realise that just putting the cards in that order does not cause them to "execute" in that order?

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

Yes Bill, job is run using ICETOOL.

Sorry, I am not very sure why VLSHRT was coded initially.

Are you saying to put include outside as below..

I am going to try this..suggest me if im wrong

Bill Woodger · Posted: Tue Jul 10, 2012 3:58 pm

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

I have tried with Just INCLUDE outside as in my code..it worked. As you suggested, for better results, i'll try yours.

SYSOUT messages for AA record

Bill Woodger · Posted: Tue Jul 10, 2012 6:36 pm

umanaga,

You really need to understand the order that things operate in in DFSORT. Kolusu posts a link from time-to-time for others in your situation. Removing the INCLUDE from the OUTFIL and using the plain one is going to save you a lot of processing. Pity I can't charge you for the savings...

My reordering of your statements does not affect the way they run. The change is in the INCLUDE happening before the SORT and SUM so that only the required records are sorted/deduped.

Seperately, you are reading from tape. Three times. How about doing one pass of the tape and extracting the raw data you need to seperate files with OUTFIL and then process those in your steps.

Still haven't seen your ICETOOL. If you show the full set of control cards, there may be other suggestions.

Skolusu · Posted: Tue Jul 10, 2012 9:41 pm

Umanaga ,

You are sorting almost 60 million records and you are doing multiple passes for the same which is quite unnecessary. Run this JCL instead of all those 3 passes.

Bill Woodger · Posted: Wed Jul 11, 2012 3:30 am

umanaga,

First, note how Kolusu coded to avoid the space problem. Where did he get the information? From your Sysout for the AC.

You were three times reading close to 60m records. Sorting them. De-duping them. Then excluding those you don't want.

If you:

Use OPTION COPY/SORT FIELDS=COPY to process the input and split it into the three types of data you want, ignoring the rest; Use OUTFIL INCLUDE/OMIT, which will not be a problem as you are not sorting;
Change each "step" to process the new specific input file, you'll no longer need the INCLUDE/OMIT as the input will only contain the correct records.

With this approach I'd guess you'd be sorting and summing fewer than 5m records instead of 180m.

That should be a littler zippier and you will not have any problems with space.

It might no longer make sense to clump it all into one ICETOOL step, but no-one says it made a great deal of sense in the first place.

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

skolusu, thanks much for the efforts in getting this coded. but not clear on one..

My AA, AB, AC records are of different length in input file. So in sort creating AA file only for its record length. But your sort job all o/p files are 673 length.

Bill, here provide the full jcl

Bill Woodger · Posted: Thu Jul 12, 2012 6:34 pm

If you are reading the file six times, to my mind there is even more reason to do something along the lines of my last post. Read the file once. Use six OUTFILs. Process the six different files.

Are you saying you are happy with how you have it?

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

Bill,
With the code in my post i am still facing space issues. As mentioned by you due to reading file 7 times (not 6 i believe).

how to code OUTFIL for different file lengths..

Edited to reflect posts removed to try to keep thread manageable.

dick scherrer · Posted: Thu Jul 12, 2012 10:11 pm

Hello,

If you write multiple output files, you would assign different SPACE parameters in the JCL - depending on the expected volume of that file.

You would build records of the appropriate length for each output file.

Bill Woodger · Posted: Fri Jul 13, 2012 12:45 am

OUTREC=(5,219)

This is where you are setting the length of the output records.

You seem to have gone back to your previous set of control cards.

You need to INCLUDE/OMIT first for the record-type you want in each step. Then you will only sort and dedupe records of the the correct type.

If you follow my earlier advice your save reading close to 300m records.

Skolusu · Posted: Fri Jul 13, 2012 3:10 am

umanaga,

Ideally I would strip off the records from the tape file and then perform individual sorts. You can club AE, AF, AG sorts as they are sorting on Similar fields and lengths.

Since you have 1 sort that does a sort on the entire 673 bytes I coded up 1 step process to handle all codes. I padded up the other fields with binary zeroes.

Bill Woodger · Posted: Fri Jul 13, 2012 5:22 am

Just finally to clear up the VLSHRT. You "needed" this originally as you were sending all the records through each sort. All those many 223-byte records were going into the sorts with larger sort keys, so would have failed without the VLSHRT.

OK, nearly finally. Note Kolusu's preference for the "strip and sort seperately". All the records on your file that are of those record-types are now being extended to 677 bytes. If those record-types represent the entire file or a large proportion of it, then you may get the space problem again. Remember that Kolusu showed you how to deal with that if you want to go with the "one step" solution.

In you ICETOOL step, with multiple sorts, you would always be exposed to a failure in one of the sorts (space for instance) requiring that the earlier sorts would be run again, or that the control cards would have to be changed for the re-run.

In a multi-step solution, you don't have that problem. More JCL to code and test, but more easy to manage.

Judging from your average record lengths, the multi-step solution I'd expect to run faster. On top of that, once the extract is done, the seperate steps can be in seperate jobs, all of which can be "kicked off" by the extract, and run in parallel, rather than serially as in your original single job.

Either way, you'll have a lot of "tape drive time" freed-up as well.

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

Bill, Thanks for mentioning on VLSHRT.

skolusu, if i know all records in my input file are AA, AB, AC, AD, AE, AF, AG can i ignore the INCLUDE at begining..as it unnessarily tries to filter on them.

Also on AB record, 48-52 column data is ignored from input file. can i get this data to o/p file without having it part of SUM fields.

Bill Woodger · Posted: Mon Jul 16, 2012 5:56 pm

If one of your record-types is not doing FIELDS=NONE in your original, then it is not going to work in one go.

I'll say it once more. Extract you data from your input files into seperate files.

Seperately SORT each.

You may want to take advantage of Kolusu having spotted that some of your record types have the same length.

umanaga · New User Joined: 09 Apr 2007 Posts: 33 Location: India

Sorry i missed to copy

Bill Woodger · Posted: Mon Jul 16, 2012 8:19 pm

Yes, we know that.

So, do it differently.

I'm not going to outline it again.

dbzTHEdinosauer · Posted: Mon Jul 16, 2012 8:36 pm

Skolusu · Posted: Mon Jul 16, 2012 9:36 pm

umanaga,

I guess you have trouble making up your mind. In one post you say you don't want to eliminate dups and in another you want to eliminate them. The best option for you is to split the file and run the individual sorts.