Concatenating/sweeping datasets based on a wildcard

dcshnier · Posted: Tue Sep 21, 2010 12:25 am

Does anyone know, if a method exists (using FILEAID; SYNCSORT; IDCAMS, REXX; CLIST, and/or basic JCL, etc...) where from a single JCL step, a concatenate/copy function can be carried out where-in one combines a number of datasets based on a wildcard (i.e. PROD.*.APPFILE.*), and copies them into a single output file. All inputs would be the same DCB.

In common terminology, I suppose one might refer to this as a 'sweep' step, that sweeps in datasets based on a common set of qualifiers.

We have an old unsupported in-house ASSEMBLER function which does this from a single step, and I am exploring replacing this with something that is better supported. I do have confidence that I could probably write something on my own using COBOL calling IDCAMS or something similar; but I first want to see if the functionality is readily available from some product that we already run. Ideally, this solution should 'not' use dynamic file allocation for the output dataset; as the customers likes to see things in the open (i.e. allocated in the JCL).

thank-you kindly for any responses.

Traveler · Posted: Tue Sep 21, 2010 12:31 am

Why the wild card? Don't you know what data, files, you are expecting? Why change something that is working? Do you have the source for the assembler?

dcshnier · Posted: Tue Sep 21, 2010 12:56 am

Thank-you for your enquiry.

The wild card is needed for a number of reasons; such as:

(1) New interfaces may start sending files that need to be combined into the process and as long as we tell them what minimal qualifiers they must create their datasets as, then our JCL will automatically take them in without requiring changes to our JCL.

(2) Our process runs on a regular (weekly) basis; but files from a given interface, are not created on a regular basis; but when created, we want to take them in.

The reasons to explore a non-ASSEMBLER approach is simply because even if the source-code still exists (which arguably it may not), there is a shortage of ASSEMBLER programmers out there; and the ASSEMBLER developers that have worked here are no longer here, or are soon to retire. One other thing that I did not mention, is that the existing ASSEMBLER process has some undesired functionality that we wish we could change.
I suppose, that even if we found a non-ASSEMBLER approach, we would not use it for the existing steps that are in place; but we would use it for any new JCL steps that are being implemented that need to carry out this type of sweep function.

regards

superk · Posted: Tue Sep 21, 2010 1:03 am

My thought would be to execute the catalog search REXX exec 'SYS1.SAMPLIB(IGGCSIRX)' and pass it your mask. Then, use that output to build a concatenation list to use for something like a TSO ALLOCATE command or for the bpxwdyn dynamic allocation routine.

dick scherrer · Posted: Tue Sep 21, 2010 1:05 am

Hello,

I'm not aware of any "magic bullet" that does this, but you could obtain a list of wild-card dataset names using adrdssu/dfdss.

This list could then be used to generate the dataset concatenation you want.

superk · Posted: Tue Sep 21, 2010 1:06 am

... or do it yourself programatically using ISPF dialog services for the wildcard search.

dcshnier · Posted: Tue Sep 21, 2010 1:19 am

Thank you Kevin and Dick for your advice; all of which I will explore if/when I decide to create my own version. I do have experience using some of the items you mentioned; other items will be things that I look forward to learning about.
I suppose if I was willing to to spread it over multiple steps, then an easy approach might be to use the LISTC results to create a //DD01 (or //SORTIN) "JCL-ready" concatanation list that could be then read into a FILEAID/SYNCSORT step using an INCLUDE statement.

regards

superk · Posted: Tue Sep 21, 2010 1:24 am

I might be wrong, but I don't think LISCAT supports that type of wildcard format.

Akatsukami · Posted: Tue Sep 21, 2010 2:12 am

dcshnier · Posted: Tue Sep 21, 2010 2:16 am

Thanks Akatsukami
I had earlier read about ADDRSSU but could not find enough documentation to see if it will do exactly what I need. I also recall that sometimes, special security access is needed to execute the program. But I will look further into it.

I response to Kevin's earlier warning, It appears that Kevn is correct (at least from my testing that I just did). The LISTCAT wildcard function is just limited to leaving the final qualifier open. I then tried the 'SYS1.SAMPLIB(IGGCSIRX)' suggestion that Kevin mentions, and that allows middle qualifiers to be searched, however, it appears to not suport the '**' double asterix. Having said that, I saw a modified version on the internet that also supports the '**' double asterix.

dcshnier · Posted: Tue Sep 21, 2010 3:25 am

hI Akatsukami

Thanks for the great tip.
I almost got ADDRSSU to work.
However, initially it had a problem finding the masked dataset inputs because the input datasets were all migrated to cartridge. Do you know how to get around this?

So after manually recalling the datasets I then re-ran the job. This time it found the masked datasets but it complained that I was missing the DELETE or RENAMEU parameter. Can I not run an ADDRSSU copy function without having to immediately delete or rename the input datasets? If not, would you have an example on how to code a rename mask in the RENAMEU paramter that I could apply to the wildcarded input set of datasets. All of my attempts are failing and the manual (which I checked), is predictably, unclear.

dick scherrer · Posted: Tue Sep 21, 2010 5:14 am

Hello,

Suggest you talk with your storage management people.

IIRC, there is a way to tell adrdssu/dfdss to bypass files that cannot be backed up for some reason (migrated, locked, etc). If all you want is the dataset names, this should work.

Akatsukami · Posted: Tue Sep 21, 2010 8:47 pm

dick scherrer · Posted: Tue Sep 21, 2010 8:54 pm

I think not your "oops". . .

Methinks TS was not paying much attention. . . Hence the little "lecture"

d