IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Is Concatenation of files essential


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jun 27, 2008 4:03 pm
Reply with quote

Hi,

I want to search for a particular expression in a group of 10 files. Is it essential for me to concatenate all the files before proceeding? The occurence is same in all the files. I am using sort to find the expression.
Back to top
View user's profile Send private message
Garry Carroll

Senior Member


Joined: 08 May 2006
Posts: 1193
Location: Dublin, Ireland

PostPosted: Fri Jun 27, 2008 4:26 pm
Reply with quote

Concatenate to process in one step or else process each file separately, I'd think. Also, depends on what you mean by
Quote:
before proceeding

Do you mean concatenate in one step and then process in another? If so, that shouldn't be necessary.


Garry.
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10872
Location: italy

PostPosted: Fri Jun 27, 2008 4:28 pm
Reply with quote

the main issue here is that the concatenation will be seen as a single file
and You will lose track of the file where the search argument will be located
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jun 27, 2008 4:29 pm
Reply with quote

By
Quote:
Before proceeding
i meant to say, that before i proceed to search for the expression, shall i concatenate the files?

Each file is having a huge bulks of records due to which the job is abending with SE37 space abend error.

I am providing the space parameter as

SPACE=(CYL,(4200,4200),RLSE)
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jun 27, 2008 4:30 pm
Reply with quote

So, Enrico you suggest me to search individual files instead of concatenating them?
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10872
Location: italy

PostPosted: Fri Jun 27, 2008 4:35 pm
Reply with quote

all depends on what the result of the search should be

I would go for
single file to understand data occurrence patterns
concatenated files to extract records for further processing
Back to top
View user's profile Send private message
Craq Giegerich

Senior Member


Joined: 19 May 2007
Posts: 1512
Location: Virginia, USA

PostPosted: Fri Jun 27, 2008 5:05 pm
Reply with quote

Is the results of the search a large number of records? Are you sorting the input file before selecting the records?
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jun 27, 2008 5:23 pm
Reply with quote

The Input files are sorted and the output file is not having large number of records when compared with the number of records in the Input files.
Back to top
View user's profile Send private message
Phrzby Phil

Senior Member


Joined: 31 Oct 2006
Posts: 1042
Location: Richmond, Virginia

PostPosted: Fri Jun 27, 2008 5:38 pm
Reply with quote

SAS note: SAS provides a simple way of knowing which file (by position number in the list) of a concatenation you are in:

EOV=variable
names a variable that SAS sets to 1 when the first record in a file in a series of concatenated files is read. The variable is set only after SAS encounters the next file. Like automatic variables, the EOV= variable is not written to the data set.
Tip: Reset the EOV= variable back to 0 after SAS encounters each boundary.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Fri Jun 27, 2008 8:27 pm
Reply with quote

Hello,

Quote:
Each file is having a huge bulks of records due to which the job is abending with SE37 space abend error.
I'm confused. . .

If you read millions of records and select a couple of hundred records matching the "expression", how does the space abend occur? Reading should not require "space" icon_confused.gif

You mention "sort", but you might not need to sort the data if you are simply scanning the file for "the expression".
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia

PostPosted: Sat Jun 28, 2008 6:38 am
Reply with quote

Hi,
you can still sort the file but as long as use the include/omit then the sort will only apply to the selected records.


Gerry
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Thu Jul 03, 2008 5:02 pm
Reply with quote

Actually, I was trying to merge the files using SYNCSORT. But the job got abended. The sysout message that I received is as under:

Code:

ALLOCATION HAS FAILED FOR ALL VOLUMES SELECTED FOR DATA SET           
&NAME..CLAIMDEC.OUT                                             
THERE ARE (732) CANDIDATE VOLUMES OF WHICH (571) ARE ENABLED OR QUIESCED
THERE WERE 1 CANDIDATE STORAGE GROUPS OF WHICH THE FIRST 1           
WERE ELIGIBLE FOR VOLUME SELECTION.                                             
THE CANDIDATE STORAGE GROUPS WERE:S1LRG90 S2LRG90 SGDISTPU                     
161 VOLUMES WERE REJECTED BECAUSE THE SMS VOLUME STATUS WAS DISABLED 
571 VOLUMES WERE REJECTED BECAUSE THEY DID NOT HAVE SUFFICIENT SPACE (041A041D)
SMS ALLOCATED TO DDNAME (SORTOUT )                         
DSN (T3709BTX.G3709T01.TBP.CLAIMDEC.OUT          )         
STORCLAS (STANDARD) MGMTCLAS (MCSTNDRD) DATACLAS (DCDEFLT)
VOL SER NOS= SML1TE                                       


Can any one please help me out in regards to the same.
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Thu Jul 03, 2008 5:03 pm
Reply with quote

The job that I am using is as under:
Code:

//SORT01   EXEC PGM=SORT                                             
//*                                                                 
//SYSPRINT DD   SYSOUT=*                                             
//SYSOUT   DD   SYSOUT=*                                             
//*                                                                 
//SORTIN01  DD DISP=SHR,DSN=&NAME..DEC2005.CMI
//SORTIN02  DD DISP=SHR,DSN=&NAME..DEC2005.CMI
//SORTIN03  DD DISP=SHR,DSN=&NAME..DEC2005.CMI
//SORTOUT   DD DSN=&NAME..CLAIMDEC.OUT,                             
//             DISP=(NEW,CATLG,CATLG),                               
//             DCB=(RECFM=FB,LRECL=5200,BLKSIZE=0,DSORG=PS),         
//             UNIT=3390,SPACE=(CYL,(4200,4200),RLSE)               
//SYSIN     DD DISP=SHR,DSN=&NAME..PULL.SRUNJCL(XXXXXXXX)   
//*                                                                 
Back to top
View user's profile Send private message
Garry Carroll

Senior Member


Joined: 08 May 2006
Posts: 1193
Location: Dublin, Ireland

PostPosted: Thu Jul 03, 2008 5:08 pm
Reply with quote

This problem is something you should take up with your storage administrators. You are failing to get sufficient space allocated via SMS.

Garry.
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA

PostPosted: Thu Jul 03, 2008 5:09 pm
Reply with quote

Typical 3390 mod 3 disk packs have 3335 cylinders of space. Asking for 4200 cylinders requires a mod 9 or higher. From the output messages, the storage pool you're asking for space from either (1) has no mod 9 devices defined, or (2) has no device with 4200 cylinders available in 5 extents or less.

Cut the space allocation to 3335 cylinders or less, and possibly use more volumes to get the same amount of space.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Thu Jul 03, 2008 9:07 pm
Reply with quote

Hello,

Are there multiple questions in this topic?

Quote:
the output file is not having large number of records when compared with the number of records in the Input files.
How much output. . .?

What is being merged that has to do with this:
Quote:
I want to search for a particular expression in a group of 10 files


If there are multiple different questions, you need to use multiple topics. We can help separate these if it is multiple issues. If these are really the same requirement, a sngle approach needs to be worked thru rather than a "shotgun" approach.

From what i see, i'm only confused icon_confused.gif
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jul 04, 2008 9:17 am
Reply with quote

Hi Dick,

This is a single requirement.

The input files(which are 30 in #) are in tapes. The number of records in each input file is around 100million. I am required to pick up particularly 1800 records from these number of input files.

For this reason previously I was trying to concatenate the files.

Can you guide me through a good approach.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Fri Jul 04, 2008 9:45 am
Reply with quote

Hello,

Will there be 1800 total records selected or will there be 1800 records per input tape selected?

1800 5k records is not a terribly high volume. . . You still might talk with your storage management peole and ask what is the proper device and/or dataclas for your output files. They will best know the storage configuration on yhour system.

How long does it take to read one of these input tapes? Without knowing much about your data and your environment, my first approach would probably be to split the task nto 5 or even 10 jobs and create a smaller selected output from each. Once the separate jobs successfully complete, the selected files could be copied back to one tape file or used in-place on dasd until this requirement to use the data is met and then backed up.

Again, coordinating with the storage management people should help.
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jul 04, 2008 10:31 am
Reply with quote

Thank you Dick for your suggestion.

Is there any method by which we can restore the Tape datasets into DASD?
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia

PostPosted: Fri Jul 04, 2008 10:46 am
Reply with quote

Hi,
why would you need to copy the tape datasets to disk, you still need to read all of the data and use up lots of disk space.


Gerry
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jul 04, 2008 10:57 am
Reply with quote

But, can we sort that huge number of input records that are residing in the tape?

The job is utilizing huge amount of CPU time and at the end its abending with SE37.
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia

PostPosted: Fri Jul 04, 2008 11:05 am
Reply with quote

Hi,

it makes no difference whether you are sorting from disk or tape.

The SE37 is on which file ?

Why are you sorting the file if you are extracting data ?

Extract the data and then sort it. From my understanding, you are not selecting many records.


Gerry
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jul 04, 2008 11:34 am
Reply with quote

The runjcl that I am using is as under:

Code:

//SORT01   EXEC PGM=SORT                                               
//*                                                                     
//SYSPRINT DD   SYSOUT=*                                               
//SYSOUT   DD   SYSOUT=*                                               
//*                                                                     
//SORTIN01  DD DISP=SHR,DSN=&NAME..SSET2.MITDPFI.DEC2005.CMI   
//SORTIN02  DD DISP=SHR,DSN=&NAME..SSET3.MITDPFI.DEC2005.CMI   
//SORTIN03  DD DISP=SHR,DSN=&NAME..SSET6.MITDPFI.DEC2005.CMI   
//SORTOUT   DD DSN=&NAME..CLAIMDEC.OUT,                                 
//             DISP=(NEW,CATLG,CATLG),                                 
//             DCB=(RECFM=FB,LRECL=5200,BLKSIZE=0,DSORG=PS),           
//             UNIT=3390,SPACE=(TRK,(3335,3335),RLSE)                   
//SYSIN     DD *
  MERGE FIELDS=(24,7,CH,A)               
  INCLUDE COND=(24,7,CH,EQ,C'0125032',OR,
                24,7,CH,EQ,C'0175002')   
  SUM FIELDS=NONE                       
/*     
//*
Back to top
View user's profile Send private message
swapnadeep.ganguly

Active User


Joined: 21 Mar 2007
Posts: 203
Location: India

PostPosted: Fri Jul 04, 2008 11:44 am
Reply with quote

I am getting the SE37 abend on the SORTOUT file.
Back to top
View user's profile Send private message
gcicchet

Senior Member


Joined: 28 Jul 2006
Posts: 1702
Location: Australia

PostPosted: Fri Jul 04, 2008 12:01 pm
Reply with quote

Hi,

like it has been mentioned earlier, run it 1 file at a time and merge at the end.

Output can be written to tape if space is an issue.

Can you also post the joblog output including messages.

Gerry
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM Goto page 1, 2  Next

 


Similar Topics
Topic Forum Replies
No new posts Write line by line from two files DFSORT/ICETOOL 7
No new posts Compare only first records of the fil... SYNCSORT 7
No new posts Merge two VSAM KSDS files into third ... JCL & VSAM 6
No new posts Joinkeys - 5 output files DFSORT/ICETOOL 7
No new posts How to append a PS file into multiple... JCL & VSAM 3
Search our Forums:

Back to Top