Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

Multiple tape inputs with huge data - Performance optimize
Goto page 1, 2  Next
 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM
View previous topic :: :: View next topic  
Author Message
JAYESHBALAN

New User


Joined: 07 Jun 2012
Posts: 4
Location: india

PostPosted: Mon Jun 11, 2012 12:46 pm    Post subject: Multiple tape inputs with huge data - Performance optimize
Reply with quote

Im using 4 tape inputs with a total of some 2.5 billion records in SORTIN. the step is using a simple inrec statement. It takes some 50 mins to run this step alone. Could someone help me reduce the runtime of this step!!!
Back to top
View user's profile Send private message

vasanthz

Global Moderator


Joined: 28 Aug 2007
Posts: 1445
Location: Azeroth

PostPosted: Mon Jun 11, 2012 12:49 pm    Post subject:
Reply with quote

Hi and welcome to the forum,
Show us the JCL step and with the SORT cards that you are using. The more relevant information you provide, the more relevant suggestions you would get.
Back to top
View user's profile Send private message
JAYESHBALAN

New User


Joined: 07 Jun 2012
Posts: 4
Location: india

PostPosted: Mon Jun 11, 2012 3:11 pm    Post subject: Please find jcl below:
Reply with quote

Jcl:
Code:
//SCPS720F EXEC PGM=SORT                                     
//*                                                         
//SYSIN    DD  DSN=DEVSVS1.CTRL.CARDS(SCPC720B),             
//             DISP=SHR                                     
//*                                                         
//SORTIN   DD  DSN=SCPX.SCPF720C.ITM.ASR.LCT.LVL1.DLT,       
//             DISP=SHR                                     
//         DD  DSN=SCPX.SCPF720E.ITM.ASR.LCT.LVL2.DLT,       
//             DISP=SHR                                     
//         DD  DSN=SCPX.SCPF720B.ITM.ASR.LCT.LVL1.IRT.UPD,   
//             DISP=SHR                                     
//         DD  DSN=SCPX.SCPF720D.ITM.ASR.LCT.LVL2.IRT.UPD,   
//             DISP=SHR                                     
//*                                                         
//SORTOUT  DD  DSN=SCPX.SCPF720F.JOBID.CHANGE.FORJDA(+1),   
//             DISP=(NEW,CATLG,DELETE),                     
//             UNIT=TAPEV,EXPDT=99000,LABEL=(1,SL)           
//*                                                         
//SYSOUT   DD  SYSOUT=*                                     
//SYSPRINT DD  SYSOUT=*

Control card:
Code:
INREC FIELDS=(1:1,1,                         * INDICATOR         
               2:C',',                                           
               3:2,4,BI,EDIT=(TTTTTTTTT),     * ASSORTMENT NUMBER
              12:C',',                                           
              13:6,2,BI,EDIT=(TTTTT),         * LOCATION NUMBER   
              18:C',',                                           
              19:8,2,BI,EDIT=(TTTTT),         * ORGANIZATION ID   
              24:C',',                                           
              25:10,2,BI,EDIT=(TTTTT),        * JOB POS CODE     
              30:C',',                                           
              31:12,4,BI,EDIT=(TTTTTTTTT),    * PERSONNEL NUMBER 
              40:C',',                                           
              41:16,4,BI,EDIT=(TTTTTTTTT))    * ITEM NUMBER       
 SORT FIELDS=(3,9,CH,A,                       * ASSORTMENT NUMBER
              13,5,CH,A,                      * LOCATION NUMBER   
              19,5,CH,A,                      * ORGANIZATION ID   
              25,5,CH,A,                      * JOB POS CODE     
             31,9,CH,A,                      * PERSONNEL NUMBER 
             41,9,CH,A)                      * ITEM NUMBER       
SUM FIELDS=NONE
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Mon Jun 11, 2012 3:21 pm    Post subject:
Reply with quote

well you have 6 binary to decimal with and edit coversions,
and you have idiotically told SORT to create 6 separate keys,
where 1 key would suffice.

you will need to provide sysout, jesmsg, and other sortmsgs for anyone to give you advice.

actually, you should bundle it all up and email it to dfsort,
so that they can tell you what you are doing
and what you could do to have it improve, if anything.
Back to top
View user's profile Send private message
JAYESHBALAN

New User


Joined: 07 Jun 2012
Posts: 4
Location: india

PostPosted: Mon Jun 11, 2012 4:26 pm    Post subject: Reply to: Multiple tape inputs with huge data - Performance
Reply with quote

I have attached the sysout of the entire job. SCPS720F is my step.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7241

PostPosted: Mon Jun 11, 2012 5:08 pm    Post subject: Reply to: Multiple tape inputs with huge data - Performance
Reply with quote

I would think that smaller records would sort faster.

If you do it on OUTREC instead of INREC you won't be sorting squintillions of commas and, as dbz has mentioned, you could have one key (though it probably doesn't make a difference, apparently).

Sort on the whole record, then your SUM, then it'll do the OUTREC processing where you can do your rearranging.

What order are your input datasets? If they happen to be in the order you want (each of them) then you could use MERGE instead of sort (four seperate input DDs). Is the SORT also so that you can do the SUM?

Tell us the full story, or follow dbz's advice.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7241

PostPosted: Mon Jun 11, 2012 6:02 pm    Post subject: Reply to: Multiple tape inputs with huge data - Performance
Reply with quote

I've looked at your output.

You have Syncsort, not DFSORT, so have posted in the wrong forum (should have been JCL) and wasted people's time.
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Mon Jun 11, 2012 10:03 pm    Post subject:
Reply with quote

Hello,

How much of the elapsed time is due to tape loads and unloads? Depending on how tapes are handled on your system (manual versus robot versus virtual tape), considerable wait time may happen.
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 7914
Location: Bellevue, IA

PostPosted: Mon Jun 11, 2012 10:42 pm    Post subject:
Reply with quote

Have you tried adding BUFNO to each tape DD statement? Having 30 to 50 buffers per data set will help performance, but may require additional region for the increased buffer space.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7241

PostPosted: Tue Jun 12, 2012 12:03 am    Post subject: Reply to: Multiple tape inputs with huge data - Performance
Reply with quote

Your input is 19 bytes, after your inrec it is 49 bytes. Which means you have 30 bytes extra which can be moved to OUTREC.

You have SUM with FIELDS=NONE, yet out of the 2.5billion records, not a single one was duplicate, suggesting that you may not need it (check the file specification/documentation).

If you can change those, it should be a bit zippier.

You seem to have a couple of earlier JOINKEYS which get no matches. There has to be a "design" way to avoid reading 150m records to not find a match.
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Tue Jun 12, 2012 1:45 am    Post subject:
Reply with quote

Hello,

Suggest you run a test that sorts the individual files with no editing.

Once the files are sorted, MERGE (all in one step) them into the new output file and edit the results in the output file.
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Tue Jun 12, 2012 4:55 am    Post subject:
Reply with quote

Hi,

I'm not able to see/download your attachment.

Why don't we ban attachments.


Gerry
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Tue Jun 12, 2012 8:38 am    Post subject: Reply to: Multiple tape inputs with huge data - Performance
Reply with quote

Hi Gerry,

If you right-click on the attachment and Save As to a .txt file it should properly align.

d
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 1717
Location: UK

PostPosted: Tue Jun 12, 2012 10:36 am    Post subject:
Reply with quote

I just ignore attachments. If domrone wants help they should help the helper - adding extra steps (downloading) is hindering not helping. I am sure attachments could be disabled (but maybe not)
Back to top
View user's profile Send private message
enrico-sorichetti

Global Moderator


Joined: 14 Mar 2007
Posts: 10202
Location: italy

PostPosted: Tue Jun 12, 2012 11:11 am    Post subject: Reply to: Multiple tape inputs with huge data - Performance
Reply with quote

and even if somebody can download/look at the attachments expecting people to wander thru 2000 lines of info is a bit too much icon_evil.gif
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1703
Location: Australia

PostPosted: Tue Jun 12, 2012 11:43 am    Post subject:
Reply with quote

Hi Dick,

thanks for the suggestion but I don't get the option to Save As.


Gerry
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 1717
Location: UK

PostPosted: Tue Jun 12, 2012 10:31 pm    Post subject:
Reply with quote

I do not know when 'someone' became 'domrone'! must have been typing in the darkness of my bedroom - especially at 5 in the morning.
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 1717
Location: UK

PostPosted: Tue Jun 12, 2012 10:34 pm    Post subject:
Reply with quote

right click gives 'save image as' which, I assume, is offering to save the icon and not the document. Maybe in the olden days....
Back to top
View user's profile Send private message
Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1738
Location: Bloomington, IL

PostPosted: Tue Jun 12, 2012 10:43 pm    Post subject:
Reply with quote

Nic Clouston wrote:
I do not know when 'someone' became 'domrone'! must have been typing in the darkness of my bedroom - especially at 5 in the morning.

Perhaps an portmanteau of "don padrone"? icon_wink.gif
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Tue Jun 12, 2012 11:27 pm    Post subject: Reply to: Multiple tape inputs with huge data - Performance
Reply with quote

Hello,

Nope, NOT save image as. . .

Quote:
If you right-click on the attachment and Save As to a .txt file it should properly align.
And i fat-fingered here. . . icon_redface.gif

It should be "Save Target As". . .

Apologies. . .

d
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM All times are GMT + 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts TAPE VERIFY utility?? MrExtraordinare JCL & VSAM 1 Fri Dec 09, 2016 10:57 pm
This topic is locked: you cannot edit posts or make replies. Unwrap the data based on delimiter X'25' bhavana yalavarthi DFSORT/ICETOOL 30 Fri Dec 09, 2016 10:25 am
No new posts Is the 'prompt' data for the DSLIST p... Willy Jensen TSO/ISPF 2 Tue Dec 06, 2016 4:38 am
This topic is locked: you cannot edit posts or make replies. How to move a long alphanumeric data ... lind sh COBOL Programming 8 Mon Dec 05, 2016 7:51 pm
No new posts PL/I code tuning/Performance improvement Virendra Shambharkar PL/I & Assembler 4 Mon Dec 05, 2016 11:57 am


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us