IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Copying large number of datasets to members


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
Jeff Almond

New User


Joined: 30 Mar 2021
Posts: 4
Location: United Kingdom

PostPosted: Tue Mar 30, 2021 7:35 pm
Reply with quote

First post here and looking for some suggestions.
I have approx 500 files that have been transferred to an MVS system. I need to copy them to PDS and considered IEBGENER. Wrapped the copy in a PROC and executed it like so:

Code:
//TOPDS    PROC DSIN=MISSING.INPUT.FILE,DSOUT=NULL                 
//GENER    EXEC PGM=IEBGENER                                       
//SYSPRINT DD SYSOUT=*                                             
//SYSIN    DD DUMMY                                                 
//SYSUT1   DD DISP=SHR,DSN=&DSIN                                   
//SYSUT2   DD DISP=SHR,                                             
//          DSN=AAAA.BBB.CCCCCC.DDDDDD.EEEEEEE.TEST(&DSOUT)         
//SYSPRINT DD SYSOUT=*                                             
//SYSIN    DD DUMMY                                                 
//         PEND                                                     
//PDS1     EXEC TOPDS,DSIN=XXXXXX.YYYYY.ZZZZZZZZ.WWWWW,DSOUT=ZZZZZZZ


This works fine for a small number of datasets but errors with too many EXEC's if I have too many.

Q. What is the best way to copy these datasets? This is a regular job so don't want to be commenting out EXEC's to run only 100 or so at a time...

Code tags added
Back to top
View user's profile Send private message
Joerg.Findeisen

Active User


Joined: 15 Aug 2015
Posts: 449
Location: Bamberg, Germany

PostPosted: Tue Mar 30, 2021 7:44 pm
Reply with quote

Consider using PDSE instead of PDS, first step.
Back to top
View user's profile Send private message
Pandora-Box

Global Moderator


Joined: 07 Sep 2006
Posts: 1586
Location: Andromeda Galaxy

PostPosted: Tue Mar 30, 2021 7:44 pm
Reply with quote

Do you want 500 physical sequential to be copied to one single PDS member?

Or a One to one mapping
Back to top
View user's profile Send private message
Jeff Almond

New User


Joined: 30 Mar 2021
Posts: 4
Location: United Kingdom

PostPosted: Tue Mar 30, 2021 7:50 pm
Reply with quote

Pandora-Box wrote:
Do you want 500 physical sequential to be copied to one single PDS member?

Or a One to one mapping


500 sequential to 500 members
Back to top
View user's profile Send private message
Pandora-Box

Global Moderator


Joined: 07 Sep 2006
Posts: 1586
Location: Andromeda Galaxy

PostPosted: Tue Mar 30, 2021 9:12 pm
Reply with quote

I would recommend for LMCOPY would come in handy I believe there should be samples in forum
Back to top
View user's profile Send private message
Jeff Almond

New User


Joined: 30 Mar 2021
Posts: 4
Location: United Kingdom

PostPosted: Tue Mar 30, 2021 9:21 pm
Reply with quote

Pandora-Box wrote:
I would recommend for LMCOPY would come in handy I believe there should be samples in forum

Thanks for the help, I'll read up on this.
Back to top
View user's profile Send private message
sergeyken
Warnings : 2

Senior Member


Joined: 29 Apr 2008
Posts: 1009

PostPosted: Wed Mar 31, 2021 2:10 am
Reply with quote

If you do not find LMCOPY to be useful for some reason, it can be also implemented in REXX, with complexity seriously below average.
Back to top
View user's profile Send private message
sergeyken
Warnings : 2

Senior Member


Joined: 29 Apr 2008
Posts: 1009

PostPosted: Wed Mar 31, 2021 2:18 am
Reply with quote

If you used a PDSE as your output library, and also split your 500-steps job into 2-50 smaller jobs (less than 255 steps each), then you might run several of your jobs in parallel. It may speed up the whole process, with minimum efforts required.
Back to top
View user's profile Send private message
Willy Jensen

Active User


Joined: 01 Sep 2015
Posts: 442
Location: Denmark

PostPosted: Wed Mar 31, 2021 2:25 am
Reply with quote

Something like this perhaps?
//ADD EXEC PGM=IEBUPDTE,PARM=NEW
//SYSPRINT DD SYSOUT=*
//SYSUT2 DD DISP=SHR,DSN=target.pds
//SYSIN DD *
./ ADD NAME=member1
// DISP=SHR,DSN=input.dsn1
./ ADD NAME=member2
// DISP=SHR,DSN=input.dsn2

and so on..
Back to top
View user's profile Send private message
sergeyken
Warnings : 2

Senior Member


Joined: 29 Apr 2008
Posts: 1009

PostPosted: Wed Mar 31, 2021 3:08 am
Reply with quote

Something is missing:
Code:

//ADD      EXEC PGM=IEBUPDTE,PARM=NEW                                       
//SYSPRINT DD SYSOUT=*                                             
//SYSUT2   DD DISP=SHR,DSN=target.pds
//SYSIN    DD *
./ ADD NAME=member1
// DD DISP=SHR,DSN=input.dsn1
// DD *
./ ADD NAME=member2
// DD DISP=SHR,DSN=input.dsn2
// DD *
./ ADD NAME=member3
. . . . and so on..


P.S.
Total number of DD statements per one single job step is also limited to some value...
Back to top
View user's profile Send private message
Joerg.Findeisen

Active User


Joined: 15 Aug 2015
Posts: 449
Location: Bamberg, Germany

PostPosted: Wed Mar 31, 2021 3:18 am
Reply with quote

sergeyken wrote:
P.S.
Total number of DD statements per one single job step is also limited to some value...

Depending on TIOT Size set in ALLOC member. Sample:
Code:
TIOT            SIZE:             64 (MAX DDS: 3273)
Back to top
View user's profile Send private message
steve-myers

Active Member


Joined: 30 Nov 2013
Posts: 881
Location: The Universe

PostPosted: Wed Mar 31, 2021 3:59 am
Reply with quote

A few thoughts.

Several people have suggested running several streams in parallel. It's dangerous and it won't really save much time even when the PDS is really a PDSE, which, in theory, can handle multiple streams adding members in parallel. Allocating the input data set is what uses the time.

The JCL only solution, similar to your initial idea, is viable, though you have to split it into several jobs because of limits on number of steps in a job. Don't worry about DD statement limits in a single step that some people have mentioned. No way you'll run into that limit.

I presume your input data sets all have a name like XXX.member.YYY. If you have even basic programming skills it should not be too terribly difficult to extract the member name portion of the data set name and prepare your copy job, whether it's JCL only or something like an IDCAMS run where you create

REPRO INDATASET( ... ) OUTDATASET( PDS(member))

The advantage of the IDCAMS solution is you can run all the REPRO commands as one ugly job step.
Back to top
View user's profile Send private message
Joerg.Findeisen

Active User


Joined: 15 Aug 2015
Posts: 449
Location: Bamberg, Germany

PostPosted: Wed Mar 31, 2021 4:05 am
Reply with quote

steve-myers wrote:
The JCL only solution, similar to your initial idea, is viable, though you have to split it into several jobs because of limits on number of steps in a job. Don't worry about DD statement limits in a single step that some people have mentioned. No way you'll run into that limit.

From my experience, at some point you will run into that limit.
Back to top
View user's profile Send private message
steve-myers

Active Member


Joined: 30 Nov 2013
Posts: 881
Location: The Universe

PostPosted: Wed Mar 31, 2021 12:47 pm
Reply with quote

Joerg.Findeisen wrote:
steve-myers wrote:
The JCL only solution, similar to your initial idea, is viable, though you have to split it into several jobs because of limits on number of steps in a job. Don't worry about DD statement limits in a single step that some people have mentioned. No way you'll run into that limit.

From my experience, at some point you will run into that limit.
IEBGENER requires 4 DD statements. My IMPORT idea allocates 2 "temporary" allocations that can be freed by allocation if IMPORT is lazy about freeing its allocations. In 50+ years I don't think I've ever run into too many DD statements, not to be confused with running out of DD DYNAM DD statements in MVT. Get real.
Back to top
View user's profile Send private message
Joerg.Findeisen

Active User


Joined: 15 Aug 2015
Posts: 449
Location: Bamberg, Germany

PostPosted: Wed Mar 31, 2021 1:15 pm
Reply with quote

@steve-myers: I have no doubts that you have never hit the limit, but others have. Never rely on hard limits even if they seem far away. Just saying.
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10763
Location: italy

PostPosted: Wed Mar 31, 2021 1:36 pm
Reply with quote

LMCOPY is not performancewise the best choice IMO
the ts would have to write anyway a rexx script to do it

I would write a rexx script to ...

read a file containing ...
source dataset name, target dataset name, target member name

for each record allocate input(sortin), output(sortout) , invoke dfsort to do the copy

the proper allocate with reuse and a free afterwards would take care of the number of allocated datasets limitations
( tested for 1000 datasets )



see here for the overall logic and the discussion of the reuse parameter
ibmmainframes.com/viewtopic.php?t=57579&highlight=alloc+reuse
and here
ibmmainframes.com/viewtopic.php?t=52353&postdays=0&postorder=asc&start=25
Back to top
View user's profile Send private message
Jeff Almond

New User


Joined: 30 Mar 2021
Posts: 4
Location: United Kingdom

PostPosted: Wed Mar 31, 2021 3:02 pm
Reply with quote

Thanks to all who took the time and replied. I'm going to go with the IDCAMS suggestion because as stated "you can run all the REPRO commands as one ugly job step" and this is really the driver. I've read up on all the other suggestions and learned things in the process so whilst I may not be using your suggestion it has been of value to me. Once again, thanks all.
Back to top
View user's profile Send private message
Pandora-Box

Global Moderator


Joined: 07 Sep 2006
Posts: 1586
Location: Andromeda Galaxy

PostPosted: Thu Apr 01, 2021 2:49 am
Reply with quote

Hi Enrico

Thanks much better approach indeed icon_smile.gif
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM

 


Similar Topics
Topic Forum Replies
No new posts Unable to retrieve Datasets Names usi... CLIST & REXX 16
No new posts REXX allocated datasets in JCL CLIST & REXX 9
No new posts COUNT the number of duplicates DFSORT/ICETOOL 3
No new posts fetch the record number from FMNMAIN ... CLIST & REXX 10
No new posts Very Large Sort failed - Seeking reco... DFSORT/ICETOOL 7
Search our Forums:

Back to Top