Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

EXCP difference in sort & idcams

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM
View previous topic :: :: View next topic  
Author Message
razesh84

New User


Joined: 05 Apr 2010
Posts: 41
Location: Kolkata,India

PostPosted: Wed Sep 22, 2010 6:47 pm    Post subject: EXCP difference in sort & idcams
Reply with quote

I've copied a large vsam(iam)[rec count : 8222316 ] file into flat file.First by repro reuse then sort(syncsort) with copy option.

Here are the run statistics
idcams :
elapsed : 19 min
cpu : 00:01:26.19
excp : 1524145

sort :
elapsed : 22 min
cpu : 00:00:47.11
excp : 360411

what strikes me is the excp counts.In both the cases the output file has the BLKSIZE=27998 & copied from the same iam file with CISIZE = 18452.

Per my knowledge excp count refers the no of blocks (of data) trasnferrred between dasd & memory.If both the files having same attribute then why there is a huge difference in excp counts?

can anyone please explain what happens here.
Let me know if you need more details.
Back to top
View user's profile Send private message

dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Wed Sep 22, 2010 7:02 pm    Post subject:
Reply with quote

just a guess,
and I am throwing it out there, expecting to get trashed,

BUT:

I thought EXCP counts were physical I/Os,
and syncsort obviously sucks-up more data per i/o than IDCAMS.
Back to top
View user's profile Send private message
razesh84

New User


Joined: 05 Apr 2010
Posts: 41
Location: Kolkata,India

PostPosted: Wed Sep 22, 2010 7:21 pm    Post subject:
Reply with quote

Quote:
I thought EXCP counts were physical I/Os,

exactly.
& i guess 1 I/O means 1 block of data transfered.

so syncsort intermediately uses a larger block for processing but the blocksize of the final file remains same with IDCAMS.

still not sure!! icon_confused.gif
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Wed Sep 22, 2010 7:35 pm    Post subject:
Reply with quote

Hello,

Quote:
excp count refers the no of blocks (of data) trasnferrred between dasd & memory
Not quite accurate. . . EXCP means EXecute Channel Program - which causes i/o.

Quote:
so syncsort intermediately uses a larger block for processing but the blocksize of the final file remains same with IDCAMS.
Syncsort often processes more than one block at a time.

For any large volume of data, Syncsort will use less i/o and less cpu than idcams.
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 7912
Location: Bellevue, IA

PostPosted: Wed Sep 22, 2010 7:39 pm    Post subject:
Reply with quote

IIRC, SSCH count is a count of physical blocks of data transferred. EXCP counts vary with buffering, chained I/O, and other factors. If your IDCAMS run used default buffering, try adding BUFNO=50 to the sequential file and AMP=('BUFND=50,BUFNI=10') to your VSAM file, then rerun the IDCAMS step.
Back to top
View user's profile Send private message
dbzTHEdinosauer

Global Moderator


Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne

PostPosted: Wed Sep 22, 2010 7:54 pm    Post subject:
Reply with quote

razesh84 wrote:
& i guess 1 I/O means 1 block of data transfered.


bad guess. i believe that a physical i/o reads a minimum of 1 track.
and if you are blocked at 3 blocks / track - that would mean 3 blocks,
if indeed the i/o transfer is 1 track.

all this is from 20 years ago. I imagine controllers su**-up more than a track-at-a-time now.
Back to top
View user's profile Send private message
nevilh

Active User


Joined: 01 Sep 2006
Posts: 258

PostPosted: Wed Sep 22, 2010 9:49 pm    Post subject:
Reply with quote

Not 100% sure but Syncsort used to use Data Chaining (99%sure) and EXCP chaining (not so sure) . This reduces the excp count considerably
Back to top
View user's profile Send private message
razesh84

New User


Joined: 05 Apr 2010
Posts: 41
Location: Kolkata,India

PostPosted: Wed Sep 22, 2010 10:49 pm    Post subject: Reply to: EXCP difference in sort & idcams
Reply with quote

thanks to all icon_biggrin.gif
yes it was a bad guess. i understand 1 I/O not refers to 1 block of data transferred.
Quote:

IIRC, SSCH count is a count of physical blocks of data transferred

can you tell me where can i find these counts?

Quote:
try adding BUFNO=50 to the sequential file and AMP=('BUFND=50,BUFNI=10') to your VSAM file, then rerun the IDCAMS step

i'll run the job with the changes & post the result.
are these AMP parameter value holds good for both VSAM & IAM.

Quote:
Not 100% sure but Syncsort used to use Data Chaining (99%sure) and EXCP chaining (not so sure) . This reduces the excp count considerably

so can i conclude if i dont use any extra buffer then sort is more faster than idcams repro
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Wed Sep 22, 2010 10:55 pm    Post subject:
Reply with quote

Hello,

I believe that even with extra buffers, Syncsort will still use less resources than idcams. . .
Back to top
View user's profile Send private message
razesh84

New User


Joined: 05 Apr 2010
Posts: 41
Location: Kolkata,India

PostPosted: Thu Sep 23, 2010 4:25 pm    Post subject:
Reply with quote

Hi,
i ran the idcams step with buffer as suggested by Robert but the job didnt show any improvement.then i ran the syncsort also with buffer,again no improvement.

idcams with buffer :
elapsed : 20 min
cpu : 00:01:42.10
excp : 1618157

sort with buffer :
elapsed : 23 min
cpu : 00:00:46.63
excp : 386613

as my input dataset is a IAM file not VSAM, i have a feeling that the buffer allocation may not be optimal.
are the calcutions for buffer(BUFNI/BUFND) same for both VSAM & IAM?
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 7912
Location: Bellevue, IA

PostPosted: Thu Sep 23, 2010 4:58 pm    Post subject:
Reply with quote

I've never worked enough with IAM to know how buffers are handled with it. The normal VSAM and sequential file defaults usually allow for big improvements in EXCP counts and run times by adding buffers, but IAM may not -- you'll need to check the manual.
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Thu Sep 23, 2010 7:02 pm    Post subject:
Reply with quote

Hello,

Syncsort does the magic dynamically/internally. . .

The thing i've seen that "helps" Syncsort is using really good block sizes for the input and out files.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts Limit duplicate records in the SORT pshongal SYNCSORT 6 Mon Nov 21, 2016 12:54 pm
No new posts How to convert the VBM file to VB or... Sulabh Agrawal JCL & VSAM 4 Fri Nov 18, 2016 1:04 pm
No new posts Sort records based on numeric field. Alks SYNCSORT 2 Wed Oct 19, 2016 10:14 pm
No new posts How to delete second instance from Fl... Gunapala CN DFSORT/ICETOOL 6 Tue Oct 18, 2016 11:42 pm
No new posts abend sort based on count records in ... anatol DFSORT/ICETOOL 5 Mon Oct 17, 2016 10:10 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us