|
View previous topic :: View next topic
|
| Author |
Message |
razesh84
New User
Joined: 05 Apr 2010
Posts: 41
Location: Kolkata,India
|
|
|
|
I've copied a large vsam(iam)[rec count : 8222316 ] file into flat file.First by repro reuse then sort(syncsort) with copy option.
Here are the run statistics
idcams :
elapsed : 19 min
cpu : 00:01:26.19
excp : 1524145
sort :
elapsed : 22 min
cpu : 00:00:47.11
excp : 360411
what strikes me is the excp counts.In both the cases the output file has the BLKSIZE=27998 & copied from the same iam file with CISIZE = 18452.
Per my knowledge excp count refers the no of blocks (of data) trasnferrred between dasd & memory.If both the files having same attribute then why there is a huge difference in excp counts?
can anyone please explain what happens here.
Let me know if you need more details. |
|
| Back to top |
|
 |
dbzTHEdinosauer
Global Moderator
Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne
|
|
|
|
just a guess,
and I am throwing it out there, expecting to get trashed,
BUT:
I thought EXCP counts were physical I/Os,
and syncsort obviously sucks-up more data per i/o than IDCAMS. |
|
| Back to top |
|
|
razesh84
New User
Joined: 05 Apr 2010
Posts: 41
Location: Kolkata,India
|
|
|
|
| Quote: |
| I thought EXCP counts were physical I/Os, |
exactly.
& i guess 1 I/O means 1 block of data transfered.
so syncsort intermediately uses a larger block for processing but the blocksize of the final file remains same with IDCAMS.
still not sure!!  |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
| Quote: |
| excp count refers the no of blocks (of data) trasnferrred between dasd & memory |
Not quite accurate. . . EXCP means EXecute Channel Program - which causes i/o.
| Quote: |
| so syncsort intermediately uses a larger block for processing but the blocksize of the final file remains same with IDCAMS. |
Syncsort often processes more than one block at a time.
For any large volume of data, Syncsort will use less i/o and less cpu than idcams. |
|
| Back to top |
|
|
Robert Sample
Global Moderator
Joined: 06 Jun 2008
Posts: 8697
Location: Dubuque, Iowa, USA
|
|
|
|
| IIRC, SSCH count is a count of physical blocks of data transferred. EXCP counts vary with buffering, chained I/O, and other factors. If your IDCAMS run used default buffering, try adding BUFNO=50 to the sequential file and AMP=('BUFND=50,BUFNI=10') to your VSAM file, then rerun the IDCAMS step. |
|
| Back to top |
|
|
dbzTHEdinosauer
Global Moderator
Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne
|
|
|
|
| razesh84 wrote: |
| & i guess 1 I/O means 1 block of data transfered. |
bad guess. i believe that a physical i/o reads a minimum of 1 track.
and if you are blocked at 3 blocks / track - that would mean 3 blocks,
if indeed the i/o transfer is 1 track.
all this is from 20 years ago. I imagine controllers su**-up more than a track-at-a-time now. |
|
| Back to top |
|
|
nevilh
Active User
Joined: 01 Sep 2006
Posts: 262
|
|
|
|
| Not 100% sure but Syncsort used to use Data Chaining (99%sure) and EXCP chaining (not so sure) . This reduces the excp count considerably |
|
| Back to top |
|
|
razesh84
New User
Joined: 05 Apr 2010
Posts: 41
Location: Kolkata,India
|
|
|
|
thanks to all 
yes it was a bad guess. i understand 1 I/O not refers to 1 block of data transferred.
| Quote: |
IIRC, SSCH count is a count of physical blocks of data transferred |
can you tell me where can i find these counts?
| Quote: |
| try adding BUFNO=50 to the sequential file and AMP=('BUFND=50,BUFNI=10') to your VSAM file, then rerun the IDCAMS step |
i'll run the job with the changes & post the result.
are these AMP parameter value holds good for both VSAM & IAM.
| Quote: |
| Not 100% sure but Syncsort used to use Data Chaining (99%sure) and EXCP chaining (not so sure) . This reduces the excp count considerably |
so can i conclude if i dont use any extra buffer then sort is more faster than idcams repro |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
I believe that even with extra buffers, Syncsort will still use less resources than idcams. . . |
|
| Back to top |
|
|
razesh84
New User
Joined: 05 Apr 2010
Posts: 41
Location: Kolkata,India
|
|
|
|
Hi,
i ran the idcams step with buffer as suggested by Robert but the job didnt show any improvement.then i ran the syncsort also with buffer,again no improvement.
idcams with buffer :
elapsed : 20 min
cpu : 00:01:42.10
excp : 1618157
sort with buffer :
elapsed : 23 min
cpu : 00:00:46.63
excp : 386613
as my input dataset is a IAM file not VSAM, i have a feeling that the buffer allocation may not be optimal.
are the calcutions for buffer(BUFNI/BUFND) same for both VSAM & IAM? |
|
| Back to top |
|
|
Robert Sample
Global Moderator
Joined: 06 Jun 2008
Posts: 8697
Location: Dubuque, Iowa, USA
|
|
|
|
| I've never worked enough with IAM to know how buffers are handled with it. The normal VSAM and sequential file defaults usually allow for big improvements in EXCP counts and run times by adding buffers, but IAM may not -- you'll need to check the manual. |
|
| Back to top |
|
|
dick scherrer
Moderator Emeritus
Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix
|
|
|
|
Hello,
Syncsort does the magic dynamically/internally. . .
The thing i've seen that "helps" Syncsort is using really good block sizes for the input and out files. |
|
| Back to top |
|
|
|
|
