|
View previous topic :: View next topic
|
| Author |
Message |
Smouch Smith
New User
Joined: 27 Oct 2021
Posts: 17
Location: Germany
|
|
|
|
I have inherited a fiarly complex REXX 'application'.
It as spread across many tens (nearly one hundred) of files, some of which are menu definitions, and some are the actual worker scripts.
These all sit within a single PDS, but then are organized into sub-PDSs, and of course, eventually seperate DSNs
PDSs
MYUSER.dev.TheApp
MYUSER.dev.TheApp.CLIST
MYUSER.dev.TheApp.DATA
MYUSER.dev.TheApp.EXEC
MYUSER.dev.TheApp.MLIB
MYUSER.dev.TheApp.PLIB
etc.
DSNs
MYUSER.dev.TheApp.PLIB.APP001 - the main panel
MYUSER.dev.TheApp.PLIB.APP002 - a sub panel
MYUSER.dev.TheApp.PLIB.APP003 - another sub panel
MYUSER.dev.TheApp.PLIB.APP005
etc..
I am fairly certain that many of the existing DSNs are junk, and not actually used. I wish to clean down (or at least identify) the unusued DSNs
How do I do this ? |
|
| Back to top |
|
 |
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2141
Location: USA
|
|
|
|
Use IDCAMS utility with series of statements like this one:
| Code: |
| DELETE ‘unused DSNAME’ |
|
|
| Back to top |
|
|
Pedro
Global Moderator
Joined: 01 Sep 2006
Posts: 2594
Location: Silicon Valley
|
|
|
|
| Quote: |
| These all sit within a single PDS |
fyi. I think you have a misunderstanding of what a PDS is. There is no such thing as a sub-PDS.
Perhaps you mean that they all have a similar naming convention. |
|
| Back to top |
|
|
Pedro
Global Moderator
Joined: 01 Sep 2006
Posts: 2594
Location: Silicon Valley
|
|
|
|
| Quote: |
| identify) the unused DSNs |
There is no easy way to do this. My suggestion is to try to exercise the panels and keep track of what is used. Rename everything else to use a different high level qualifier and wait until somebody complains. Restore the datasets to original names as required. But this is pretty disruptive. Keep the archived files for several years.
If you can, maybe create a way to log the usage. At the beginning of each rexx and clist, update a log file with date, userid, and rexx program name. Maybe also log when a panel is about to be displayed. |
|
| Back to top |
|
|
sergeyken
Senior Member
Joined: 29 Apr 2008
Posts: 2141
Location: USA
|
|
|
|
| Pedro wrote: |
| Quote: |
| identify) the unused DSNs |
There is no easy way to do this. My suggestion is to try to exercise the panels and keep track of what is used. Rename everything else to use a different high level qualifier and wait until somebody complains. Restore the datasets to original names as required. But this is pretty disruptive. Keep the archived files for several years.
If you can, maybe create a way to log the usage. At the beginning of each rexx and clist, update a log file with date, userid, and rexx program name. Maybe also log when a panel is about to be displayed. |
This is a general problem for any large system. Not only for used/unused REXX/CLIST libraries, but even more serious, for multiple load modules spread over LINK/LOAD libraries during the life of the company. In most cases the administration gives up, and leaves everything as is, for future generations.
In one big government organization I was working about 8 years there was a list of 97 load libraries in the system LINKLIST, created by many system programmers in the past 40 or 50 years. There is practically no chance to distinguish between used, and never-used modules for the time being. |
|
| Back to top |
|
|
Pedro
Global Moderator
Joined: 01 Sep 2006
Posts: 2594
Location: Silicon Valley
|
|
|
|
| Quote: |
| created by many system programmers in the past |
I had such an issue 30 years ago... for compiled programs, make sure you have the source code. Try to re-compile everything every so many years. |
|
| Back to top |
|
|
Smouch Smith
New User
Joined: 27 Oct 2021
Posts: 17
Location: Germany
|
|
|
|
I was thinking that some kind of script could be used.
In Linux, this wouldn't be very difficult to write. In fact, it's a one-liner:
for file in $(find . -type f); do grep -rL $(basename $file); done
This at least identifies the files that do not contain the text matching the names of files in all subdirectories.
Can't something like that be done ? |
|
| Back to top |
|
|
Garry Carroll
Senior Member
Joined: 08 May 2006
Posts: 1205
Location: Dublin, Ireland
|
|
|
|
Again, as Pedro stated earlier, PDS does not have sub-PDSes.
Also, while you can scan libraries for members that contain specified text (the others therefore not having that text), I fail to see how that can assist you. Finding text that references another REXX member is no guarantee that the member is/was ever executed.
Garry. |
|
| Back to top |
|
|
Smouch Smith
New User
Joined: 27 Oct 2021
Posts: 17
Location: Germany
|
|
|
|
Yeah, I understand that the PDS thing isn't exacly the same as a directory, with sub-directories, but, that's the way it's being used here.
It would be good enough for now to identify DSNs which are definitely not used - they are definitely not used if they are never referenced by any panel or other REXX in this app.
So, I need a way to iterate over all the DSNs in this App, and for each DSN bare name, I want to find out if that is ever referenced in any other DSN within the App. If not, I think I can safely delete (or at least rename it in preparation for deletion) |
|
| Back to top |
|
|
Willy Jensen
Active Member
Joined: 01 Sep 2015
Posts: 734
Location: Denmark
|
|
|
|
| "they are definitely not used if they are never referenced by any panel or other REXX in this app" is not neccessarily true, the name could be generated dynamically. |
|
| Back to top |
|
|
Pedro
Global Moderator
Joined: 01 Sep 2006
Posts: 2594
Location: Silicon Valley
|
|
|
|
| Quote: |
| if they are never referenced |
They could be called from within some rexx program or panel in a differently named data set. |
|
| Back to top |
|
|
Smouch Smith
New User
Joined: 27 Oct 2021
Posts: 17
Location: Germany
|
|
|
|
| Quote: |
| the name could be generated dynamically. |
| Quote: |
| They could be called from within some rexx program or panel in a differently named data set. |
I'm not really concerned about such possibilities. I will deal with any special circumstance later. I really want to start by clearing the "Unused" REXX, so that I have a better picture of the magnitude of the App. I have a complete backup in case I really do discover such usage paradigms. |
|
| Back to top |
|
|
Willy Jensen
Active Member
Joined: 01 Sep 2015
Posts: 734
Location: Denmark
|
|
|
|
The job below does a any-to-any crossref. It builds a list of member for all requested libraries and then scan all those libraries for the names. The job uses the PDSMAIN / PDS86 utility from your textwww.cbttape.org file 182. If you have something else, you must modify the statements accordingly. It will probably run for a veeeery long time. You can do some member name filtering in the 1st step.
The final list will look like this:
MTM4P01 in WJENSEN.LIB.EXEC MTM
| Code: |
//*
//X1 EXEC PGM=IEFBR14
//PDSCMD DD UNIT=SYSDA,DISP=(MOD,DELETE),DSN=XXXXXX.PDSCMD,
// RECFM=FB,LRECL=80,BLKSIZE=0,SPACE=(TRK,(5,5))
//*
//A EXEC RXINSTRM
//REXX DD *
/* load library list */
"execio * diskr liblist (stem lib. finis)"
do n=1 to lib.0
lib.n=strip(lib.n)
end
/* make member list */
mbrn=0
do libn=1 to lib.0
zz=outtrap('ml.')
"listds '"lib.libn"' members"
zz=outtrap('off')
say 'Listing' lib.libn 'returns' ml.0 -7 'names'
do n=7 to ml.0
m=strip(ml.n)
if length(m)<5 then iterate /* filter */
parse value mbrn+1 m with mbrn mbr.mbrn
end
end
mbr.0=mbrn
say '# members before sort:' mbr.0
/* note, 'sort' must be in lowercase */
call bpxwunix 'sort -u', 'MBR.', 'MBR.', 'STDERR.' /* unique */
say '# members after sort :' mbr.0
/* generate PDSMAIN */
do ln=1 to lib.0
if ln=1 then queue "PDSMAIN '"lib.ln"'"
else queue "C '"lib.ln"'"
do mn=1 to mbr.0
queue " FIND : '"mbr.mn"'"
end
end
queue 'END'
"execio" queued() "diskw pdscmd (finis)"
exit 0
XMsg: if arg(1)<>'' then say arg(1);return word(arg(2) 0,1)
//PDSCMD DD UNIT=SYSDA,DISP=(,CATLG),DSN=XXXXXX.PDSCMD,
// RECFM=FB,LRECL=80,BLKSIZE=0,SPACE=(TRK,(5,5))
//LIBLIST DD *
XXXXXX.LIB.ISPPLIB
XXXXXX.LIB.EXEC
//*
//X2 EXEC PGM=IEFBR14
//SYSOUT DD UNIT=SYSDA,DISP=(MOD,DELETE),DSN=XXXXXX.SYSOUT,
// RECFM=VBA,LRECL=240,BLKSIZE=0,SPACE=(TRK,1)
//*
//B EXEC PGM=IKJEFT1B
//STEPLIB DD DISP=SHR,DSN=WJENSEN.LIB.LOADLIB *PDSMAIN*
//SYSTSIN DD DISP=SHR,DSN=XXXXXX.PDSCMD
//SYSTSPRT DD UNIT=SYSDA,DISP=(,CATLG),DSN=XXXXXX.SYSOUT,
// RECFM=VBA,LRECL=240,BLKSIZE=0,SPACE=(CYL,(5,5))
//*
//C EXEC RXINSTRM
//LIST DD DISP=SHR,DSN=XXXXXX.SYSOUT
//REXX DD *
/* list list, generate condensed list */
parse value '' with dsn mbr
do forever
"Execio 1000 diskr list (stem lst.)"
if lst.0=0 then leave
do ln=1 to lst.0
l=substr(lst.ln,2) /* drop asa */
select
when substr(l,1,8)=='PDSMAIN ',
| substr(l,1,6)=='PDS86 ' ,
| substr(l,1,6)=='C ' then,
dsn=strip(word(l,2),"b","'")
when substr(l,1,8)==' FIND : ' then,
mbr=strip(word(l,3),"b","'")
when substr(l,1,8)=='** FIND ' then,
say left(mbr,8) 'in' dsn word(l,3)
otherwise nop
end
end
end
"Execio 0 diskr list (finis)"
//*
|
|
|
| Back to top |
|
|
Pete Wilson
Active Member
Joined: 31 Dec 2009
Posts: 592
Location: London
|
|
|
|
| Do you have FDR in your site? Let me know, I can give you some code to get the last-ref dates on the PDS's and members themselves. |
|
| Back to top |
|
|
Pedro
Global Moderator
Joined: 01 Sep 2006
Posts: 2594
Location: Silicon Valley
|
|
|
|
| Quote: |
| It builds a list of member for all requested libraries and then scan all those libraries for the names. |
Hmmm, you use PDSMAIN... I think it would make this process more consumable if you built a SRCHFOR job. |
|
| Back to top |
|
|
steve-myers
Active Member
Joined: 30 Nov 2013
Posts: 917
Location: The Universe
|
|
|
|
Many programs are available to list data set attributes. For example -
| Code: |
(DSLIST 02/22/22 13.06) D S L I S T F O R B A T C H 25 FEB 2022 17:25 PAGE 1
VOLUME RECFM LRECL BLKSZ DSORG CREDT XPIRDT LASTREF ALLOC USED DATA SET NAME
XXXSTG FB 80 27920 PS 22JAN22 ----- 22FEB22 2 2 ZZZZZZ.DSLIST.ASM
XXX002 VBM 137 27998 PS 22JAN22 ----- 22FEB22 6 5 ZZZZZZ.DSLIST.ASMLIST
XXX003 FB 80 3200 PO 16JAN22 ----- 24JAN22 13 6 ZZZZZZ.DSLIST.XXX.PDS
XXX006 VB 255 3120 PS 18OCT21 ----- 22FEB22 1 1 ZZZZZZ.DSLIST.CLIST
XXX003 FB 80 27920 PO 26JAN22 ----- 26JAN22 15 1 ZZZZZZ.DSLIST.EVALUATE.PDS
XXXSTG U 0 32760 PO 22JAN22 ----- 25FEB22 3 3 ZZZZZZ.DSLIST.LOAD
XXX002 FB 80 27920 PS 22JAN22 ----- 22FEB22 1 1 ZZZZZZ.DSLIST.OBJ
XXX003 VB 4100 27998 PS 22JAN22 ----- 22FEB22 19 5 ZZZZZZ.DSLIST.PCLIST
XXXSTG VBA 125 27998 PS 22JAN22 ----- 25FEB22 1 1 ZZZZZZ.DSLIST.SYSPRINT
XXX003 VB 4100 27998 PS 22JAN22 ----- 10FEB22 1 1 ZZZZZZ.DSLIST.TEXT
XXX006 FBA 121 27951 PS 24JAN22 ----- 22FEB22 1 1 ZZZZZZ.DSLIST.UPDATE.SYSPRINT
XXX006 FB 80 3120 PS 21JAN22 ----- 21JAN22 1 1 ZZZZZZ.DSLIST.V10A.UPDATE.ASM
XXXSTG FB 80 3120 PS 22JAN22 ----- 22FEB22 1 1 ZZZZZZ.DSLIST.V11.UPDATE.ASM
65 29 44% |
Most of these programs, like this one, "file" 1025 in the CBT collection, display the last reference date for the data set.
There is no practical way to obtain a last reference date for a member of a PDS. Members with directory entries that have ISPF statisticss will show the last date the member was updated. |
|
| Back to top |
|
|
Willy Jensen
Active Member
Joined: 01 Sep 2015
Posts: 734
Location: Denmark
|
|
|
|
| @Pedro "I think it would make this process more consumable if you built a SRCHFOR job" you have a point, I'll give it a try. I just happen to like PDS86 a lot. |
|
| Back to top |
|
|
Smouch Smith
New User
Joined: 27 Oct 2021
Posts: 17
Location: Germany
|
|
|
|
| Willy Jensen wrote: |
| The job below does a any-to-any crossref. It builds a list of member for all requested libraries and then scan all those libraries for the names..... |
Thank you for the response.
I must admit, after reading through this, I cannot decide if I want to laugh or cry.
It seems hard to believe that a simpler solution can't be found.
I can't really understand what is being done here anyway. What do I need to change to provide my list of PDSs ? I don't see where I provide the top-level level namespace for iteration.
[/quote] |
|
| Back to top |
|
|
Garry Carroll
Senior Member
Joined: 08 May 2006
Posts: 1205
Location: Dublin, Ireland
|
|
|
|
| Quote: |
| I can't really understand what is being done here anyway. What do I need to change to provide my list of PDSs ? I don't see where I provide the top-level level namespace for iteration. |
PDS and namespace are from different environments. There is no top-level other than the PDS itself.
A REXX program (A) in PDS#1 can invoke a REXX program (B) in PDS#2 which can invoke a REXX program(C) in PDS#1. REXX programs don't have to be in a PDS - REXX in a PS dataset can invoke a REXX member in a PDS & vice-versa.
Garry. |
|
| Back to top |
|
|
Smouch Smith
New User
Joined: 27 Oct 2021
Posts: 17
Location: Germany
|
|
|
|
| Garry Carroll wrote: |
A REXX program (A) in PDS#1 can invoke a REXX program (B) in PDS#2 which can invoke a REXX program(C) in PDS#1. REXX programs don't have to be in a PDS - REXX in a PS dataset can invoke a REXX member in a PDS & vice-versa.
Garry. |
I understand that. I am using "top level" in a logical context only.
Still, I wish I could understand how this seemingly trivial search (see my linux BASH one-liner above) can possibly require a hundred line script. |
|
| Back to top |
|
|
Willy Jensen
Active Member
Joined: 01 Sep 2015
Posts: 734
Location: Denmark
|
|
|
|
| The script I provided will do a cross-ref between libraries in the //LIBLIST DD libref. You need to create a list of all membernames in all libraries, then scan all the libraries for those names. I'm not a Linux person, but I dont see how your one-liner does that job. Or I might have misunderstood something. |
|
| Back to top |
|
|
Smouch Smith
New User
Joined: 27 Oct 2021
Posts: 17
Location: Germany
|
|
|
|
| Willy Jensen wrote: |
| ... but I dont see how your one-liner does that job. Or I might have misunderstood something. |
# find every object of type file (-type f) within the current directory (.)
# and subdirectories (default dehavior)
for file in $(find . -type f);
# search for text (grep) recursively (-r) BUT, only show results (a file name)
# if the search text is not found (-L), where the search text is the file name
# (stripping directory name) discovered by 'find',
do grep -rL $(basename $file);
# self explanitory
done
NOTE: This is in effect 3 lines, as you can see by the use of semicolon. But, since each line is quite short, it's common (and considered good style) to place such a loop on a single line. |
|
| Back to top |
|
|
enrico-sorichetti
Superior Member
Joined: 14 Mar 2007
Posts: 10888
Location: italy
|
|
|
|
while the oneliner produces the proper output according to the used syntax
the output will be just useless
rexxa containing panelA
rexxb containing panelB
both will apear in the output
for portability across different grep implementations it woud be better to write
for file in $(find . -type f); do grep -rL $(basename $file) * ; done
tested on fedora and apple big sur
with both the gnu and bsd versions of grep
( in both cases grep will search every file in the current directory structure ) |
|
| Back to top |
|
|
Smouch Smith
New User
Joined: 27 Oct 2021
Posts: 17
Location: Germany
|
|
|
|
| enrico-sorichetti wrote: |
the output will be just useless
rexxa containing panelA
rexxb containing panelB
|
I don't understand what you mean by this.
When the bash script is executed, the only output is the names of files which do not appear in other files - IOW, those files which are present for apparently no reason, they are 'junk' files cluttering up the application namespace.
| Code: |
# create a sample environment
$ mkdir test
$ cd test
$ mkdir dir1
$ mkdir dir2
$ mkdir dir3
# create some files
$ touch dir1/file001
$ touch dir1/file002
$ touch dir1/file003
$ touch dir2/file001
$ touch dir2/file004
$ touch dir3/file002
$ touch dir3/file004
$ touch dir3/file005
# populate a few files with other files names
$ echo file001 >> dir1/file003
$ echo file002 >> dir1/file003
$ echo file002 >> dir2/file004
$ echo file002 >> dir3/file004
$ echo file003 >> dir3/file004
$ echo file004 >> dir1/file003
# at this point the text 'file005' does not exist in any of the files created
# grep with -rL to show how that works - all the files should be listed
$ grep -rL 'file005' .
./dir1/file001
./dir1/file002
./dir1/file003
./dir2/file001
./dir2/file004
./dir3/file002
./dir3/file004
./dir3/file005
# search for a filename that does exist as text within some of the files
# those files should not appear in the result list
$ grep -rL 'file002' .
./dir1/file001
./dir1/file002
./dir2/file001
./dir3/file002
./dir3/file005
# Thus file002 is referenced, but file005 is not |
So, in fact, my one-liner doesn't produce perfectly clean results, but it would be trivial to check that the list of files which does not contain file005 is the same size as the list of all files. |
|
| Back to top |
|
|
enrico-sorichetti
Superior Member
Joined: 14 Mar 2007
Posts: 10888
Location: italy
|
|
|
|
my point was not about the grep in se,
but using it inside the one-liner
if you are more comfortable using unix like tools
why not copy the datasets involved to the USS side an do your research from there
( natuarally depending on the amount of data to be copied ) |
|
| Back to top |
|
|
|
|
