IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Multiple tape inputs with huge data - Performance optimize


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
JAYESHBALAN

New User


Joined: 07 Jun 2012
Posts: 4
Location: india

PostPosted: Mon Jun 11, 2012 12:46 pm
Reply with quote

Im using 4 tape inputs with a total of some 2.5 billion records in SORTIN. the step is using a simple inrec statement. It takes some 50 mins to run this step alone. Could someone help me reduce the runtime of this step!!!
Back to top
View user's profile Send private message
vasanthz

Global Moderator


Joined: 28 Aug 2007
Posts: 1744
Location: Tirupur, India

PostPosted: Mon Jun 11, 2012 12:49 pm
Reply with quote

Hi and welcome to the forum,
Show us the JCL step and with the SORT cards that you are using. The more relevant information you provide, the more relevant suggestions you would get.
Back to top
View user's profile Send private message
JAYESHBALAN

New User


Joined: 07 Jun 2012
Posts: 4
Location: india

PostPosted: Mon Jun 11, 2012 3:11 pm
Reply with quote

Jcl:
Code:
//SCPS720F EXEC PGM=SORT                                     
//*                                                         
//SYSIN    DD  DSN=DEVSVS1.CTRL.CARDS(SCPC720B),             
//             DISP=SHR                                     
//*                                                         
//SORTIN   DD  DSN=SCPX.SCPF720C.ITM.ASR.LCT.LVL1.DLT,       
//             DISP=SHR                                     
//         DD  DSN=SCPX.SCPF720E.ITM.ASR.LCT.LVL2.DLT,       
//             DISP=SHR                                     
//         DD  DSN=SCPX.SCPF720B.ITM.ASR.LCT.LVL1.IRT.UPD,   
//             DISP=SHR                                     
//         DD  DSN=SCPX.SCPF720D.ITM.ASR.LCT.LVL2.IRT.UPD,   
//             DISP=SHR                                     
//*                                                         
//SORTOUT  DD  DSN=SCPX.SCPF720F.JOBID.CHANGE.FORJDA(+1),   
//             DISP=(NEW,CATLG,DELETE),                     
//             UNIT=TAPEV,EXPDT=99000,LABEL=(1,SL)           
//*                                                         
//SYSOUT   DD  SYSOUT=*                                     
//SYSPRINT DD  SYSOUT=*

Control card:
Code:
INREC FIELDS=(1:1,1,                         * INDICATOR         
               2:C',',                                           
               3:2,4,BI,EDIT=(TTTTTTTTT),     * ASSORTMENT NUMBER
              12:C',',                                           
              13:6,2,BI,EDIT=(TTTTT),         * LOCATION NUMBER   
              18:C',',                                           
              19:8,2,BI,EDIT=(TTTTT),         * ORGANIZATION ID   
              24:C',',                                           
              25:10,2,BI,EDIT=(TTTTT),        * JOB POS CODE     
              30:C',',                                           
              31:12,4,BI,EDIT=(TTTTTTTTT),    * PERSONNEL NUMBER 
              40:C',',                                           
              41:16,4,BI,EDIT=(TTTTTTTTT))    * ITEM NUMBER       
 SORT FIELDS=(3,9,CH,A,                       * ASSORTMENT NUMBER
              13,5,CH,A,                      * LOCATION NUMBER   
              19,5,CH,A,                      * ORGANIZATION ID   
              25,5,CH,A,                      * JOB POS CODE     
             31,9,CH,A,                      * PERSONNEL NUMBER 
             41,9,CH,A)                      * ITEM NUMBER       
SUM FIELDS=NONE
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Mon Jun 11, 2012 3:21 pm
Reply with quote

well you have 6 binary to decimal with and edit coversions,
and you have idiotically told SORT to create 6 separate keys,
where 1 key would suffice.

you will need to provide sysout, jesmsg, and other sortmsgs for anyone to give you advice.

actually, you should bundle it all up and email it to dfsort,
so that they can tell you what you are doing
and what you could do to have it improve, if anything.
Back to top
View user's profile Send private message
JAYESHBALAN

New User


Joined: 07 Jun 2012
Posts: 4
Location: india

PostPosted: Mon Jun 11, 2012 4:26 pm
Reply with quote

I have attached the sysout of the entire job. SCPS720F is my step.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Mon Jun 11, 2012 5:08 pm
Reply with quote

I would think that smaller records would sort faster.

If you do it on OUTREC instead of INREC you won't be sorting squintillions of commas and, as dbz has mentioned, you could have one key (though it probably doesn't make a difference, apparently).

Sort on the whole record, then your SUM, then it'll do the OUTREC processing where you can do your rearranging.

What order are your input datasets? If they happen to be in the order you want (each of them) then you could use MERGE instead of sort (four seperate input DDs). Is the SORT also so that you can do the SUM?

Tell us the full story, or follow dbz's advice.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Mon Jun 11, 2012 6:02 pm
Reply with quote

I've looked at your output.

You have Syncsort, not DFSORT, so have posted in the wrong forum (should have been JCL) and wasted people's time.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Mon Jun 11, 2012 10:03 pm
Reply with quote

Hello,

How much of the elapsed time is due to tape loads and unloads? Depending on how tapes are handled on your system (manual versus robot versus virtual tape), considerable wait time may happen.
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8700
Location: Dubuque, Iowa, USA

PostPosted: Mon Jun 11, 2012 10:42 pm
Reply with quote

Have you tried adding BUFNO to each tape DD statement? Having 30 to 50 buffers per data set will help performance, but may require additional region for the increased buffer space.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Tue Jun 12, 2012 12:03 am
Reply with quote

Your input is 19 bytes, after your inrec it is 49 bytes. Which means you have 30 bytes extra which can be moved to OUTREC.

You have SUM with FIELDS=NONE, yet out of the 2.5billion records, not a single one was duplicate, suggesting that you may not need it (check the file specification/documentation).

If you can change those, it should be a bit zippier.

You seem to have a couple of earlier JOINKEYS which get no matches. There has to be a "design" way to avoid reading 150m records to not find a match.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Tue Jun 12, 2012 1:45 am
Reply with quote

Hello,

Suggest you run a test that sorts the individual files with no editing.

Once the files are sorted, MERGE (all in one step) them into the new output file and edit the results in the output file.
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia

PostPosted: Tue Jun 12, 2012 4:55 am
Reply with quote

Hi,

I'm not able to see/download your attachment.

Why don't we ban attachments.


Gerry
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Tue Jun 12, 2012 8:38 am
Reply with quote

Hi Gerry,

If you right-click on the attachment and Save As to a .txt file it should properly align.

d
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK

PostPosted: Tue Jun 12, 2012 10:36 am
Reply with quote

I just ignore attachments. If domrone wants help they should help the helper - adding extra steps (downloading) is hindering not helping. I am sure attachments could be disabled (but maybe not)
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10886
Location: italy

PostPosted: Tue Jun 12, 2012 11:11 am
Reply with quote

and even if somebody can download/look at the attachments expecting people to wander thru 2000 lines of info is a bit too much icon_evil.gif
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia

PostPosted: Tue Jun 12, 2012 11:43 am
Reply with quote

Hi Dick,

thanks for the suggestion but I don't get the option to Save As.


Gerry
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK

PostPosted: Tue Jun 12, 2012 10:31 pm
Reply with quote

I do not know when 'someone' became 'domrone'! must have been typing in the darkness of my bedroom - especially at 5 in the morning.
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK

PostPosted: Tue Jun 12, 2012 10:34 pm
Reply with quote

right click gives 'save image as' which, I assume, is offering to save the icon and not the document. Maybe in the olden days....
Back to top
View user's profile Send private message
Akatsukami

Global Moderator


Joined: 03 Oct 2009
Posts: 1787
Location: Bloomington, IL

PostPosted: Tue Jun 12, 2012 10:43 pm
Reply with quote

Nic Clouston wrote:
I do not know when 'someone' became 'domrone'! must have been typing in the darkness of my bedroom - especially at 5 in the morning.

Perhaps an portmanteau of "don padrone"? icon_wink.gif
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19243
Location: Inside the Matrix

PostPosted: Tue Jun 12, 2012 11:27 pm
Reply with quote

Hello,

Nope, NOT save image as. . .

Quote:
If you right-click on the attachment and Save As to a .txt file it should properly align.
And i fat-fingered here. . . icon_redface.gif

It should be "Save Target As". . .

Apologies. . .

d
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 2454
Location: Hampshire, UK

PostPosted: Wed Jun 13, 2012 11:16 am
Reply with quote

No 'save target as' on my system (firefox) but left click and it gives the option to run or save.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM

 


Similar Topics
Topic Forum Replies
No new posts db2 vs static data COBOL Programming 1
No new posts combine multiple unique records into ... DFSORT/ICETOOL 2
No new posts External data queue (changes?) CLIST & REXX 0
No new posts Pull data using date difference betwe... DB2 6
No new posts fast data scrambling PL/I & Assembler 10
Search our Forums:

Back to Top