IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

File matching functionality in Easytrieve to an equivalent D
IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic Quick References
View previous topic :: View next topic  
Author Message
elixir1986
Warnings : 1
New User


Joined: 10 Nov 2015
Posts: 44
Location: USA
PostPosted: Sun Aug 14, 2022 6:42 am
 Reply with quote
I am converting file matching functionality in Easytrieve to an equivalent DFSORT with JOINKEYS.

Reference used - IBM DFSORT Tuning Guide - www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R3sc236882/$file/icet100_v2r3.pdf

I converted the input signed Packed Decimal(PD) to unsigned packed decimal in DFSORT JOINKEYS step with "INREC OVERLAY".

Inputs are sorted. I have included "SORTED,NOSEQCK" commands in the SYSIN for the SORT.

Along with that I have also used HIPRMAX=0,DSPSIZE=MAX, and MOSIZE=MAX.

The DFSORT is taking about 4 CPU seconds more that Easytrieve.



Quote:
Is there any additional steps I can take to ensure improvement in the CPU time consumption?


Let me know if any additional information is needed.

Code:

OPTION HIPRMAX=0,DSPSIZE=MAX,MOSIZE=MAX
SORT FIELDS=COPY                                 
JOINKEYS FILE=XXX1,FIELDS=(80,8,A),SORTED,NOSEQCK   
JOINKEYS FILE=XXX2,FIELDS=(80,8,A),SORTED,NOSEQCK   
REFORMAT FIELDS=(F2:1,38,F1:6,5)                 
INREC OVERLAY=(80,08,PD,TO=PDF,88:08,PD,TO=PDF)
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1255
Location: Bamberg, Germany
PostPosted: Sun Aug 14, 2022 11:06 am
 Reply with quote
Try
Code:
HIPRMAX=OPTIMAL
Back to top
View user's profile Send private message
elixir1986
Warnings : 1
New User


Joined: 10 Nov 2015
Posts: 44
Location: USA
PostPosted: Sun Aug 14, 2022 11:16 am
 Reply with quote
Manual at https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R3sc236878/$file/icea100_v2r3.pdf says,

"Hipersorting can cause a small CPU time increase. When CPU optimization is a concern, you can use HIPRMAX=0 to suppress Hipersorting."

I have already overridden the HIPRMAX to 0.
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1255
Location: Bamberg, Germany
PostPosted: Sun Aug 14, 2022 12:44 pm
 Reply with quote
And difference in consumption / runtime is at your specific setup?

Maybe a non-JOINKEYS variant could also help to speed up things.
Back to top
View user's profile Send private message
elixir1986
Warnings : 1
New User


Joined: 10 Nov 2015
Posts: 44
Location: USA
PostPosted: Sun Aug 14, 2022 3:56 pm
 Reply with quote
No difference whatsoever.

Non Joinkeys variant in Sort? Or like using cobol? I’m not familiar with former.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2023
Location: USA
PostPosted: Sun Aug 14, 2022 7:30 pm
 Reply with quote
elixir1986 wrote:
I converted the input signed Packed Decimal(PD) to unsigned packed decimal in DFSORT JOINKEYS step with "INREC OVERLAY".

Let me know if any additional information is needed.

Code:

OPTION HIPRMAX=0,DSPSIZE=MAX,MOSIZE=MAX
SORT FIELDS=COPY                                 
JOINKEYS FILE=XXX1,FIELDS=(80,8,A),SORTED,NOSEQCK   
JOINKEYS FILE=XXX2,FIELDS=(80,8,A),SORTED,NOSEQCK   
REFORMAT FIELDS=(F2:1,38,F1:6,5)                 
INREC OVERLAY=(80,08,PD,TO=PDF,88:08,PD,TO=PDF)


1. The syntax of INREC OVERLAY=… seems to result in “Syntax Error”, isn’t it?

2. The statement REFORMAT is executed before INREC.
(Output from REFORMAT is used as input to INREC)
So, the result of REFORMAT is 38+5=43 bytes.
If so, it is not clear: what should be converted by INREC OVERLAY at positions 80, and 88?
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3053
Location: NYC,USA
PostPosted: Sun Aug 14, 2022 9:08 pm
 Reply with quote
How many records are processed?

Btw,You can use an INREC statement in
JNF1CNTL and/or JNF2CNTL to normalize the keys for the F1 file and/or F2.
Back to top
View user's profile Send private message
elixir1986
Warnings : 1
New User


Joined: 10 Nov 2015
Posts: 44
Location: USA
PostPosted: Sun Aug 14, 2022 9:30 pm
 Reply with quote
Rohit Umarjikar wrote:
How many records are processed?

Btw,You can use an INREC statement in
JNF1CNTL and/or JNF2CNTL to normalize the keys for the F1 file and/or F2.


Approximate record counts
SORTJNF1 - 70 Million
SORTJNF2 - 20 Million
SORTOUT - 20 Million

All the records(matching keys) from SORTJNF2 are present in SORTJNF1. From the previous Production Easytrieve run logs, I see this has been the case as well.

I have tried using INREC OVERLAY=(80,08,PD,TO=PDF,88:08,PD,TO=PDF) in JNF1CNTL but still no improvement in the CPU performance.
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1255
Location: Bamberg, Germany
PostPosted: Sun Aug 14, 2022 9:59 pm
 Reply with quote
Since you are using REFORMAT, I would go with Rohit and reduce the amount of data in INREC of JNF1CNTL and JNF2CNTL.
Back to top
View user's profile Send private message
elixir1986
Warnings : 1
New User


Joined: 10 Nov 2015
Posts: 44
Location: USA
PostPosted: Sun Aug 14, 2022 10:39 pm
 Reply with quote
How do I reduce the amount of data? I get that from a feed.

Did you mean splitting file corresponding to SORTJNF1?
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1255
Location: Bamberg, Germany
PostPosted: Sun Aug 14, 2022 11:19 pm
 Reply with quote
Basically you need to rebuild (or reduce) the records into Key (80,8) for each, plus F2:1,38 in JNF2CNTL and F1:6,5 in JNF1CNTL. You have to adjust the JOINKEYS FIELDS for this, and conversion to format PDF can be done once join processing has completed.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2023
Location: USA
PostPosted: Sun Aug 14, 2022 11:43 pm
 Reply with quote
I did not receive any response to my previous questions… There are obvious errors which nobody takes care about.

Now I can suggest: in both //JFN1CNTL and //JFN2CNTL to truncate records in INREC to only those fields needed for subsequent JOINKEYS, and REFORMAT.
It can significantly reduce the total size of data to be processed, and potentially speed-up the whole process.
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3053
Location: NYC,USA
PostPosted: Mon Aug 15, 2022 6:23 pm
 Reply with quote
Try BI format on keys along with trimming the data in subtask than in the maintask. If nothing works then work with site support to see if any of installation options are impacting this or can be changed to further time it.
www.ibm.com/docs/en/zos/2.3.0?topic=efficiently-tuning-main-storage
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2023
Location: USA
PostPosted: Mon Aug 15, 2022 10:02 pm
 Reply with quote
elixir1986 wrote:
I converted the input signed Packed Decimal(PD) to unsigned packed decimal in DFSORT JOINKEYS step with "INREC OVERLAY".

Let me know if any additional information is needed.

Code:

OPTION HIPRMAX=0,DSPSIZE=MAX,MOSIZE=MAX
SORT FIELDS=COPY                                 
JOINKEYS FILE=XXX1,FIELDS=(80,8,A),SORTED,NOSEQCK   
JOINKEYS FILE=XXX2,FIELDS=(80,8,A),SORTED,NOSEQCK   
REFORMAT FIELDS=(F2:1,38,F1:6,5)                 
INREC OVERLAY=(80,08,PD,TO=PDF,88:08,PD,TO=PDF)


One more time:
Code:
 . . . . . . . .             
 INREC OVERLAY=(80,08,PD,TO=PDF,88:08,PD,TO=PDF)
                                     *
INREC STATEMENT        : SYNTAX ERROR


Next step: REFORMAT FIELDS=(F2:1,38,F1:6,5)
will produce the record of 38+5=43 bytes.

This 43-bytes records is passed to INREC OVERLAY=(80,08,PD,TO=PDF,88:08,PD,TO=PDF)
which definitely will give the error like this one: "position specified is out of the record field", because both positions 80, and 88 are greater than 43.
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2023
Location: USA
PostPosted: Mon Aug 15, 2022 10:21 pm
 Reply with quote
My suggestion is
Code:

//SYSIN    DD  *
 OPTION HIPRMAX=0,DSPSIZE=MAX,MOSIZE=MAX
 JOINKEYS FILE=XXX1,FIELDS=(1,8,A),SORTED,NOSEQCK   
 JOINKEYS FILE=XXX2,FIELDS=(1,8,A),SORTED,NOSEQCK   
 REFORMAT FIELDS=(F2:9,38,F1:9,5)                 
 SORT FIELDS=COPY                                 
//*
//JFN1CNTL DD  *
 INREC BUILD=(80,8,PD,TO=PDF,LENGTH=8,
               6,5)
//*
//JFN2CNTL DD  *
 INREC BUILD=(80,8,PD,TO=PDF,LENGTH=8,
               1,38)
//*


Also remains unclear: why to convert PD to PDF? This field is not used in the final output record?........
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 

Similar Topics
Topic Forum Replies
No new posts Compare 2 files and retrive records f... DFSORT/ICETOOL 3
No new posts FTP VB File from Mainframe retaining ... JCL & VSAM 8
No new posts Extract the file name from another fi... DFSORT/ICETOOL 6
No new posts How to split large record length file... DFSORT/ICETOOL 10
No new posts Extracting Variable decimal numbers f... DFSORT/ICETOOL 17
Search our Forums:

Back to Top
IBMMainframes.com is not an official and/or affiliated with IBM® in anyway
Board Rules | FAQ | Downloads | Wiki | SiteMap | Contact Us