Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

Improve performance of file comparison
Goto page 1, 2  Next
 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM
View previous topic :: :: View next topic  
Author Message
Priyanka Pyne

New User


Joined: 09 Feb 2008
Posts: 95
Location: India

PostPosted: Thu Sep 29, 2011 9:27 pm    Post subject: Improve performance of file comparison
Reply with quote

Hi,

I have created one tool using REXX and JCL which will compare two mainframe files in copybook layout.I am using filemanager to compare the files. But itis taking long time to give the output result. Is there any way which can expedite the comparison?
Back to top
View user's profile Send private message

enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Thu Sep 29, 2011 9:34 pm    Post subject: Reply to: Improve performance of file comparison
Reply with quote

Quote:
But it is taking long time to give the output result.

BIBSD on... define the issue in measurable terms
long time/short time good performance/bad performarce are pretty useless terms of comparison
and usually depend on which side the opponents stand

but ... I really do not see how we could advice about it... given the scarce info You provided..
the fact that You used REXX to build the jcl is irrelevant

it is a good habit to <debug> by steps
what happened when You submitted the same compare with a hand crafted jcl ?
Back to top
View user's profile Send private message
Priyanka Pyne

New User


Joined: 09 Feb 2008
Posts: 95
Location: India

PostPosted: Thu Sep 29, 2011 9:51 pm    Post subject:
Reply with quote

I tried with #50 k data and it is taking 20 mins time to complete the comparison.
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Thu Sep 29, 2011 11:47 pm    Post subject:
Reply with quote

Hello,

What happens when you compare the files with SUPERC?
Back to top
View user's profile Send private message
Priyanka Pyne

New User


Joined: 09 Feb 2008
Posts: 95
Location: India

PostPosted: Fri Sep 30, 2011 1:13 am    Post subject:
Reply with quote

SuperC also taking almost same amount of time and moreover it does not provide the output result in Copybook layout and my requirement is to have the output result in copybook layout.
Back to top
View user's profile Send private message
Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1738
Location: Bloomington, IL

PostPosted: Fri Sep 30, 2011 1:36 am    Post subject:
Reply with quote

Priyanka Pyne wrote:
I tried with #50 k data and it is taking 20 mins time to complete the comparison.

What are the LRECLs of your data sets? What does your JCL look like?
Back to top
View user's profile Send private message
anatol

New User


Joined: 20 May 2010
Posts: 99
Location: canada

PostPosted: Fri Sep 30, 2011 1:37 am    Post subject:
Reply with quote

I found sort is better, when you split to nodups, dups. Nodups - not match, dups are the same
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Fri Sep 30, 2011 1:59 am    Post subject:
Reply with quote

Hello,

With only 50k records, the superc takes nearly 20 minutes also ?

One guess is that there is something horribly wrong with the definition of the file(s). Are both files in sequence before the compare is run?

I periodically compare several million records of 12k record length and this only takes a few minutes - less than 10 even on a bad day.
Back to top
View user's profile Send private message
Priyanka Pyne

New User


Joined: 09 Feb 2008
Posts: 95
Location: India

PostPosted: Fri Sep 30, 2011 2:01 am    Post subject:
Reply with quote

LRECL is 2094.

JCL looks like
Code:
//FILEMGR  EXEC PGM=FMNMAIN,TIME=MAXIMUM                           
//STEPLIB  DD DISP=SHR,DSN=SYS1.FILEMNGR.SFMNMOD1                 
//         DD DISP=SHR,DSN=SYSC090.COBOLZOS.PROD.SIGYCOMP         
//*FMNCOB  DD DUMMY     Uncomment to force use of FM COBOL Compiler
//*SYSPRINT DD SYSOUT=*                                           
//*.SYSPRINT.ORD.USERNME.TEST.SYSPRINT                           
//SYSPRINT DD DSN=ORD.USERNME.TEST.SYSPRINT,                     
//         SPACE=(CYL,(900,900),RLSE),                             
//         DISP=(,CATLG,DELETE),                                   
//         DCB=*.SORT1.SORTIN                                     
//FMNTSPRT DD SYSOUT=*                                             
//SYSTERM  DD SYSOUT=*                                             
//SYSIN    DD *                                                   
$$FILEM DSCMP TYPE=FORMATTED, 
$$FILEM DSNOLD=FILE1,
$$FILEM DSNOLD=FILE2                                   
/*                                             
//*.FMINSOUT.ORD.&USERNME.TEST.INSERTED.T     
//FMINSOUT DD DSN=ORD.USERNME.TEST.INSERTED.T,
//         SPACE=(CYL,(900,900),RLSE),         
//         DISP=(,CATLG,DELETE),               
//         DCB=*.SORT1.SORTIN                   


Before comparing I am sorting both the files.

I have copied only the comparison step here.
Back to top
View user's profile Send private message
Priyanka Pyne

New User


Joined: 09 Feb 2008
Posts: 95
Location: India

PostPosted: Fri Sep 30, 2011 2:03 am    Post subject:
Reply with quote

Hi Dick,

Are you refering to SuperC?

I cannot use SuperC as I need the putput result in copybook layout.
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Fri Sep 30, 2011 2:09 am    Post subject:
Reply with quote

Hello,

Yes, i understand that you don't want the superc format of the data. I was trying to understand why the superc took about the same amount of time as the filemanager run (unless i have misunderstood this).

I would expect the filemanager run to use significantly more time than superc but if they use a similar amount it may help to learn why.

All of the field level formatting takes a lot of time, but i still have no idea why 50k records takes 20 minutes. How much cpu time does each run use? EXCPs?
Back to top
View user's profile Send private message
Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1738
Location: Bloomington, IL

PostPosted: Fri Sep 30, 2011 2:12 am    Post subject:
Reply with quote

Priyanka Pyne wrote:
Hi Dick,

Are you refering to SuperC?

I cannot use SuperC as I need the putput result in copybook layout.

But a test run using SuperC will provide information useful for diagnosis. If a SuperC compare of the data also takes 20 minutes, there is a problem with the data or data set. If it only takes 20 milliseconds, then there is a problem with the File Mangler control cards.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Fri Sep 30, 2011 2:31 am    Post subject:
Reply with quote

what is the blocksize for FILE1 and FILE2? Which by the way you don't bother to provide

why are you forcing DCB parms on output files to be same as some other existing file DCB=*.SORT1.SORTIN
which, since it is not a SORT output, it is probably not optimized.

and by the way, what do you mean by:
Quote:
compare two mainframe files in copybook layout


and if there is a copybook format involved, where is the reference?
Back to top
View user's profile Send private message
Priyanka Pyne

New User


Joined: 09 Feb 2008
Posts: 95
Location: India

PostPosted: Fri Sep 30, 2011 3:19 am    Post subject:
Reply with quote

The reason being i am using this JCL as a backend of a REXX tool. Where one can provide any type of input file. And hence lenth of the output file can not be pre determind. So I am copying the DCB of the input file.

By refering Copybook layout I wanted to say the layout of the file.

I am copying the control card which I have used for this purposes.

Code:
$$FILEM DSCMP TYPE=FORMATTED,             
$$FILEM PACK=UNPACK,                     
$$FILEM SYNCH=KEYED,                     
$$FILEM KEYLOCOLD=1,                     
$$FILEM KEYLOCNEW=1,                     
$$FILEM KEYLEN=9,                         
$$FILEM KEYTYPE=CHAR,                     
$$FILEM LIST=LONG,                       
$$FILEM WIDE=YES,                         
$$FILEM HILIGHT=YES,                     
$$FILEM CHNGDFLD=YES,                     
$$FILEM IGNLEN=YES,                       
$$FILEM EXCLUDE=(,,MATCHED,),             
$$FILEM NUMDIFF=ALL,                     
$$FILEM DSNOLD=xxx.TEST.VSAM.FILE1,   
$$FILEM TCOLD=SYS2.xxxx.COPYLIB(zzzz),
$$FILEM LANG=COBOL,                       
$$FILEM SKIPOLD=0,                       
$$FILEM CMPOLD=ALL,                       
$$FILEM TCNEW=SYS2.xxxx.COPYLIB(zzzz),
$$FILEM SKIPNEW=0,                       
$$FILEM CMPNEW=ALL,                       
$$FILEM DSNNEW=xxx.TEST.VSAM.FILE2   


Previously I had provided wrong information about the data count. I am really sorry for that. The files have almost #686721 data. Because of which I was not able to open it view mode and not got the exact count.
Back to top
View user's profile Send private message
prino

Active Member


Joined: 07 Feb 2009
Posts: 984
Location: Oostende, Belgium

PostPosted: Fri Sep 30, 2011 1:56 pm    Post subject:
Reply with quote

Priyanka Pyne wrote:
... The files have almost #686721 data. Because of which I was not able to open it view mode and not got the exact count.

And obviously you've never heard of Browse, which can open files with zillions of records (if you have sithloads of time ;) )
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 1715
Location: UK

PostPosted: Fri Sep 30, 2011 2:00 pm    Post subject:
Reply with quote

Or read the sort counts from the sort step?
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Fri Sep 30, 2011 2:15 pm    Post subject:
Reply with quote

Quote:
xxx.TEST.VSAM.FILE2


are these vsam or qsam files?
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Fri Sep 30, 2011 9:08 pm    Post subject:
Reply with quote

Hello,

Just to see what happens with superc, i compared vb files with more than 8 million records:

Code:
                      LINE COMPARE SUMMARY AND STATISTICS         
                                                                   
                                                                   
                                                                   
 8243650 NUMBER OF LINE MATCHES               0  TOTAL CHANGES (PAIR
       0 REFORMATTED LINES                    0  PAIRED CHANGES (REF
       0 NEW FILE LINE INSERTIONS             0  NON-PAIRED INSERTS
       0 OLD FILE LINE DELETIONS              0  NON-PAIRED DELETES
 8243650 NEW FILE LINES PROCESSED                                   
 8243650 OLD FILE LINES PROCESSED                                   
                                                                   
LISTING-TYPE = DELTA      COMPARE-COLUMNS =    1:1065      LONGEST-LINE = 1065


And this took:
Code:
     12.82 MINUTES EXECUTION TIME
Back to top
View user's profile Send private message
Priyanka Pyne

New User


Joined: 09 Feb 2008
Posts: 95
Location: India

PostPosted: Mon Oct 03, 2011 9:42 pm    Post subject:
Reply with quote

Hi Dick,

Thanks for the SuperC result but as I mentioned earlier I cannot use SuperC as I need the comparison result in a file/copybook layout.
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10203
Location: italy

PostPosted: Mon Oct 03, 2011 9:43 pm    Post subject: Reply to: Improve performance of file comparison
Reply with quote

then You will have to bear the larger resource consumption and the longer elapsed time
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM All times are GMT + 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts PL/I code tuning/Performance improvement Virendra Shambharkar PL/I & Assembler 4 Mon Dec 05, 2016 11:57 am
No new posts Add PD field from 2nd file to PD in 1st Sushant Garje DFSORT/ICETOOL 6 Thu Dec 01, 2016 4:32 pm
No new posts File Aid to File Manager conversion murali3955 IBM Tools 4 Thu Nov 24, 2016 3:41 pm
No new posts How to convert the VBM file to VB or... Sulabh Agrawal JCL & VSAM 4 Fri Nov 18, 2016 1:04 pm
No new posts CICS Roll back partially - Need to re... dwijadas CICS 4 Wed Nov 16, 2016 4:30 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us