Joined: 29 Jun 2006 Posts: 1436 Location: Bangalore,India
Hi All,
Could it be possible to increase the # of conditions handled by INCLUDE statement? Im looking at 70 to 100 thousand conditions.
I had seen MINCORE option (I guess its a syncsort option) in other shop. I had come to know that it increases the # of conditions handled by included statement. Is the same possible in DFSORT (as my shop uses only DFSORT).
If yes, please let me know the same and also correct me if Im wrong in this regard.
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
Syncsort has a default storage value for certain operations and requires MINCORE to increase the limit. DFSORT does not have such a default and allows the maximum number of conditions automatically without the need for a parameter. But you can't use 70 to 100 thousand INCLUDE conditions regardless of how much storage is available. The limit is much, much smaller than that.
You might be able to do what you want using SPLICE with a selection file and master file instead.
Joined: 29 Jun 2006 Posts: 1436 Location: Bangalore,India
In addition to my post above,
Please tell me whether sort handles 125M records (5 flat files of each 25M records). Since the layout is similar for all the 5 files, Im in an idea to concatenate all the 5 files in SORTIN statement.
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
Hello,
I believe IFTHEN is a future feature for Syncsort. It is not in the documentation downloaded a couple of months ago.
If you have 70-100k conditions, a better solution would be designing some kind of external "rule" file(s) and processing these rules against your input file.
This is another example of something best not done in a sort even if you find a way (IMHO).
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
Dick,
Your reference to Syncsort seems irrelevant considering that the original poster said:
Quote:
my shop uses only DFSORT
Quote:
If you have 70-100k conditions, a better solution would be designing some kind of external "rule" file(s) and processing these rules against your input file.
This is another example of something best not done in a sort even if you find a way (IMHO).
Maybe or maybe not. As I said previously
Quote:
You might be able to do what you want using SPLICE with a selection file and master file instead.
That might, in fact, be a good solution using DFSORT.
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
Hi Frank,
Yup,
Quote:
Your reference to Syncsort seems irrelevant considering that the original poster said:
Obviously, i was focusing on
Quote:
I am also facing the same issue & my shop uses syncsort
and
Quote:
MINCORE option (I guess its a syncsort option)
and missed the bit about that shop being DFSORT
I believe the SPLICE "selection file and master file" approach is the same thing i was referring to by "rules". What i believe should not be done is to code 50-100k INCLUDES.
Joined: 29 Jun 2006 Posts: 1436 Location: Bangalore,India
Frank,
Thanks all for your inputs.
My selection file as well as Master file layout are provided below -
Length - 267; Key length - 14 starts @ 9 column.
Quote:
It depends on what your conditions look like. Give an example.
I dint get the above, but I had given an example below -
Sample conditions:
Code:
include cond=(9,14,CH,EQ,C'1CR0701U035A15',OR,
9,14,CH,EQ,C'19A0701C162IEH',OR,
9,14,CH,EQ,C'1PY0701U039O19',OR,
9,14,CH,EQ,C'1750701C162C05',OR, and so on)
As suggested by you, I had tried IFTHEN technique for 79997 conditions. Fortunately or unfortunately, I had the following storage error.
JESYSMSG
Code:
IEF237I JES2 ALLOCATED TO
IEA995I SYMPTOM DUMP OUTPUT
SYSTEM COMPLETION CODE=80A REASON CODE=00000010
TIME=06.53.13 SEQ=06877 CPU=0040 ASID=0044
PSW AT TIME OF ERROR 070C1000 81485F06 ILC 2 INTC 0D
Joined: 23 Nov 2006 Posts: 19244 Location: Inside the Matrix
Hello,
The 80A is because the step needed more memory than it could get (which may be found in the "MVS System Codes" manual, which is linked to this site). If you look there, you will see exactly what the problem was by the reason-code that follows the 80A.
Rather than 80k includes, you might put the 80k "include values" in a file and then user a 2-file match/merge to keep/discard the records you want to "include" or "exclude.
If you look in the forum you will see several examples that Frank has provided to do this.
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
Murali,
For a 14-byte CH field, you can use about 1360 conditions per INCLUDE statement (or WHEN operand). It really isn't practical to do 80000 conditions with IFTHEN statements - it takes too much storage. A better approach would be to use the SELECT or SPLICE operator of DFSORT's ICETOOL. If you want me to show you how, I just need you to tell me whether or not there are duplicates in the input file (that is, can 1CR0701U035A15 or any other key appear in more than one record of the input file)?
Joined: 29 Jun 2006 Posts: 1436 Location: Bangalore,India
Actually my master file has duplicate values and I tried the Create files with matching and non-matching records from SORTTRCK pdf. But my peers have not agreed for it saying 'Control cards are not easily understandable for those who have not worked much with sort techniques'. So, I was wondering the same could be achieved in someother way (through INCLUDE etc ....).
Anyway Thanks one and all everyone for your valuable inputs.
Joined: 15 Feb 2005 Posts: 7129 Location: San Jose, CA
Quote:
But my peers have not agreed for it saying 'Control cards are not easily understandable for those who have not worked much with sort techniques'.
This seems to me to be equivalent to saying "we aren't capable of learning new things, even when they can help us do our job". I guess your peers wouldn't have liked the IFTHEN solution either (if it were feasible) since that's something new as well.
Why bother to ask us here how to achieve this; why not just ask your peers to solve it since they seem to have veto power over anything we could propose.
.jpg)
