Portal | Manuals | References | Downloads | Info | Programs | JCLs | Master the Mainframes
IBM Mainframe Computers Forums Index
 
Register
 
IBM Mainframe Computers Forums Index Mainframe: Search IBM Mainframe Forum: FAQ Memberlist Usergroups Profile Log in to check your private messages Log in
 

 

Syncsort - Sort a file with 300 Milllion records LRECL= 4800

 
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM
View previous topic :: :: View next topic  
Author Message
kitchu84

New User


Joined: 02 Dec 2006
Posts: 33
Location: chennai

PostPosted: Thu Jun 16, 2011 8:22 am    Post subject: Syncsort - Sort a file with 300 Milllion records LRECL= 4800
Reply with quote

I am running SYNCSORT on an i/p file with 350 million recs and LRECL of 4800 , using the DYNALLOC option without any additional SORTWK in the JCL. Below are the parameters that i am passing;

Code:
PARM='DYNALLOC=(SYSDA,230),HIPRMAX=OPTIMAL,VSCORE=16M,VSCORET=64M'


After an hour of execution i am seeing the following messages in my job spool, though my job is still running. We have addtional DASD added prior to this execution.

Can anyone please let me know what paramters or additional steps needs to be added to sort such a big file .
Does adding additional SORTWK in JCL going to help out?
Code:
------------------------------------------------------
SORT FIELDS=(09,30,CH,A,41,04,CH,A,47,30,CH,A)                                         
SYSDIAG= 165571, 3386707, 3386707, 7989825                                     
106,476K BYTES OF VIRTUAL STORAGE AVAILABLE, MAX REQUESTED,                     
   0 BYTES RESERVE REQUESTED, 102,432K BYTES USED                               
20K BYTES OF EMERGENCY SPACE ALLOCATED                                         
SORTIN   : RECFM=VB   ; LRECL=  4772; BLKSIZE= 32760                           
SORTOUT  : RECFM=VB   ; LRECL=  4772; BLKSIZE= 32760                           
98,288K BYTES OF VIRTUAL STORAGE AVAILABLE ABOVE THE 16MEG LINE,               
   0 BYTES RESERVE REQUESTED, 98,264K BYTES USED                               
G=5029,B=13672,SEGLEN=15728,BIAS=94                                             
0 PREALLOCATED SORTWORK TRACKS, 1,725,000 DYNAMICALLY ALLOCATED,               
   9,662,400 ACQUIRED IN 3,224 SECONDARY EXTENTS, 0 RELEASED, TOTAL OF 1,370,006 TRACKS USED
SORT INTERNAL ERROR - RECOVERY ATTEMPT IN PROGRESS                             
SYNCSMF  CALLED BY SYNCSORT; RC=0000                                           
SYNCSORT GLOBAL DSM SUBSYSTEM ACTIVE                                           
------------------------------------------------------------------


Edited: Please use BBcode when You post some code/error, that's rather readable, Thanks... Anuj
Back to top
View user's profile Send private message

dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Thu Jun 16, 2011 8:40 am    Post subject:
Reply with quote

Hello,

How did the job end?

How much of the data is in sequence before this sort begins? If this process is to take a smaller amount of unsorted data and sort it with the large amount, it would possibly (probably) be better to sort the small amount of data and then MERGE it with the large amount.
Back to top
View user's profile Send private message
kitchu84

New User


Joined: 02 Dec 2006
Posts: 33
Location: chennai

PostPosted: Thu Jun 16, 2011 10:49 am    Post subject: Reply to: Syncsort - Sort a file with 300 Milllion records L
Reply with quote

The job was still running , with the same above message . Hence i cancelled the job and resubmitted with the below parm and no extra SORTWK

PARM='DYNALLOC=(SYSDA,500),VSCORET=128M'

But after some 30 to 45 mins still getting the same message in the spool. Any thoughts or suggestions which would make this sort work is really appreciated.
Back to top
View user's profile Send private message
kitchu84

New User


Joined: 02 Dec 2006
Posts: 33
Location: chennai

PostPosted: Thu Jun 16, 2011 11:01 am    Post subject: Reply to: Syncsort - Sort a file with 300 Milllion records L
Reply with quote

To add to that , i need to sort this whole file based on the key and then load it back to the data base. Hence splitting and running would be tha last option that i would choose.


Any suggestions/thoughts ,
Back to top
View user's profile Send private message
PeterHolland

Global Moderator


Joined: 27 Oct 2009
Posts: 2422
Location: Netherlands, Amstelveen

PostPosted: Thu Jun 16, 2011 11:21 am    Post subject:
Reply with quote

Run the HISTOGRM program and try to sort with the information generated by HISTOGRM.

Did you try using MAXSORT?
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Thu Jun 16, 2011 11:52 am    Post subject:
Reply with quote

Split the file. Sort the splits. Merge the sorted splits.
Back to top
View user's profile Send private message
PeterHolland

Global Moderator


Joined: 27 Oct 2009
Posts: 2422
Location: Netherlands, Amstelveen

PostPosted: Thu Jun 16, 2011 11:54 am    Post subject:
Reply with quote

Bill Woodger wrote:
Split the file. Sort the splits. Merge the sorted splits.


Bill,

that is what MAXSORT will do.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Thu Jun 16, 2011 12:04 pm    Post subject:
Reply with quote

Ah, without all the fiddly bits like doing the JCL yourself. Nice. Learn something every day. Thanks Peter.

Kitchu84, definitely contact Syncsort.

When posting sort messages here, please keep the message numbers.
Back to top
View user's profile Send private message
Anuj Dhawan

Senior Member


Joined: 22 Apr 2006
Posts: 6258
Location: Mumbai, India

PostPosted: Thu Jun 16, 2011 4:25 pm    Post subject:
Reply with quote

You are using OPTION HIPRMAX, so what value for FILSZ= being used?

And, when you are dealing with some vendor-product - it's always good to include the message-ID for the error messages, which is the only thing you share with us from your problem per se.
Back to top
View user's profile Send private message
Bill Woodger

DFSORT Moderator


Joined: 09 Mar 2011
Posts: 7223

PostPosted: Thu Jun 16, 2011 4:30 pm    Post subject: Reply to: Syncsort - Sort a file with 300 Milllion records L
Reply with quote

Just as a tease, you're not doing this just to get a count of the records are you? (It is just a tease, I can see you are not)
Back to top
View user's profile Send private message
Anuj Dhawan

Senior Member


Joined: 22 Apr 2006
Posts: 6258
Location: Mumbai, India

PostPosted: Thu Jun 16, 2011 5:28 pm    Post subject:
Reply with quote

On the other hand, increasing the number in
Quote:
'DYNALLOC=(SYSDA,500),
is not going to change anything, on the fly. How much such storage allocation can be secured for the a Job is governed by many factors and before we go in to the details of DATACLAS or ACS routines at your shop, there is a default limit set by SyncSort-vendors at the time of installation itself - beyond which you can't allocate sort work file dynamically.
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 7905
Location: Bellevue, IA

PostPosted: Thu Jun 16, 2011 5:41 pm    Post subject:
Reply with quote

350 million 4800 byte records requires about 1.6 terabytes (1600 gigabytes) so you absolutely have to work with your site support group on this sort. You probably also need to contact Syncsort to get some help on how to structure this sort.

Even though you are reluctant to do so, I suspect your only option will turn out to be split the file into some number of smaller files, sort each of them seperately, then merge back into one big file so you can load your data base. And don't expect the process to be fast -- I wouldn't be surprised to see the sort elapsed time to be days, not hours, even if you do manage to sort the one big file.
Back to top
View user's profile Send private message
Anuj Dhawan

Senior Member


Joined: 22 Apr 2006
Posts: 6258
Location: Mumbai, India

PostPosted: Thu Jun 16, 2011 5:58 pm    Post subject:
Reply with quote

I concur with Robert on the advice of getting in touch with the Support Group. You also show that you are using VSCORE=16M,VSCORET=64M, these are some of those parameters which should not be used without consulting the SyncSort representativesa and might cause more damage than they cure, if not used intelligently.
Back to top
View user's profile Send private message
PeterHolland

Global Moderator


Joined: 27 Oct 2009
Posts: 2422
Location: Netherlands, Amstelveen

PostPosted: Thu Jun 16, 2011 6:02 pm    Post subject:
Reply with quote

The TS is sorting VB records. So he should run HISTOGRM and use the
HISTOGRM information to run a MAXSORT.
But then im repeating myself.
Back to top
View user's profile Send private message
Anuj Dhawan

Senior Member


Joined: 22 Apr 2006
Posts: 6258
Location: Mumbai, India

PostPosted: Thu Jun 16, 2011 6:07 pm    Post subject:
Reply with quote

I'm listening, Peter! icon_wink.gif
Back to top
View user's profile Send private message
expat

Global Moderator


Joined: 14 Mar 2007
Posts: 8593
Location: Back in jolly old England

PostPosted: Thu Jun 16, 2011 6:18 pm    Post subject:
Reply with quote

Peter, is there an echo in here icon_lol.gif
Back to top
View user's profile Send private message
PeterHolland

Global Moderator


Joined: 27 Oct 2009
Posts: 2422
Location: Netherlands, Amstelveen

PostPosted: Thu Jun 16, 2011 6:22 pm    Post subject:
Reply with quote

Guys, guys,

im not the TS so please stop making jokes on me. Beside that if you count all the times you did repeat answers then there is a lot of echoing here. icon_evil.gif
Back to top
View user's profile Send private message
Anuj Dhawan

Senior Member


Joined: 22 Apr 2006
Posts: 6258
Location: Mumbai, India

PostPosted: Fri Jun 17, 2011 3:19 pm    Post subject:
Reply with quote

echo!
Back to top
View user's profile Send private message
dick scherrer

Site Director


Joined: 23 Nov 2006
Posts: 19270
Location: Inside the Matrix

PostPosted: Fri Jun 17, 2011 8:03 pm    Post subject:
Reply with quote

Hi Guys,

RedundancyOn


Doesn't Syncsort have a tool called HISTOGRM that would be of use here?

RedundancyOff


heh heh heh icon_rolleyes.gif

Have a great weekend!

d
Back to top
View user's profile Send private message
PeterHolland

Global Moderator


Joined: 27 Oct 2009
Posts: 2422
Location: Netherlands, Amstelveen

PostPosted: Fri Jun 17, 2011 10:07 pm    Post subject:
Reply with quote

dick scherrer wrote:
Hi Guys,

RedundancyOn


Doesn't Syncsort have a tool called HISTOGRM that would be of use here?

RedundancyOff


heh heh heh icon_rolleyes.gif

Have a great weekend!

d


Not after this input Dick.
Now i have to prolong my friday happy hours the whole week-end.
But whatever, have a nice week-end too. And after some reflection the rest of the (I(C)T) community here too.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic    IBMMAINFRAMES.com Support Forums -> JCL & VSAM All times are GMT + 6 Hours
Page 1 of 1

 

Search our Forum:

Similar Topics
Topic Author Forum Replies Posted
No new posts syncsort: copy lines after the keyword shreya19 SYNCSORT 1 Fri Dec 02, 2016 9:47 am
No new posts Add PD field from 2nd file to PD in 1st Sushant Garje DFSORT/ICETOOL 6 Thu Dec 01, 2016 4:32 pm
No new posts File Aid to File Manager conversion murali3955 IBM Tools 4 Thu Nov 24, 2016 3:41 pm
No new posts Limit duplicate records in the SORT pshongal SYNCSORT 6 Mon Nov 21, 2016 12:54 pm
No new posts How to convert the VBM file to VB or... Sulabh Agrawal JCL & VSAM 4 Fri Nov 18, 2016 1:04 pm


Facebook
Back to Top
 
Mainframe Wiki | Forum Rules | Bookmarks | Subscriptions | FAQ | Tutorials | Contact Us