Joined: 28 Aug 2007 Posts: 1742 Location: Tirupur, India
Hi,
We have a SAS job, that sorts 80000000(8 followed by 7 0's) observations.
The SORTPARM for the job is
Code:
SORTPARM='DYNALLOC=(SYSALLDA,250),FILSZ=E40000000' (4 followed by 7 0's)
The job sometimes fails, it has the below messages 6 times,
Code:
+ICE097I 1 JOBNAME1.S1 SORT RECOVERING FROM B37 ABEND ON SORTWK DATA SET
IEC030I B37-04,IFG0554A,JOBNAME1,SAS,SORTWKB2,554C,PUB528,0427041D,XXXXX.YYYYY.RA000.JOBNAME1.R0142402
and then fails with
Code:
BPXP018I THREAD 2686630000000002, IN PROCESS 65729, ENDED
The saslog has
Code:
ERROR: THE SAS SYSTEM STOPPED PROCESSING THIS STEP BECAUSE OF INSUFFICIENT MEMORY
Could you please let me know if the failure is related to FILSZ value being much smaller than actual number of records, or due to MEMORY shortage.
Joined: 09 Mar 2011 Posts: 7309 Location: Inside the Matrix
I'd say yes. 100% out is too much. If DFSORT is not able to estimate the records itself (because it is not processing the files itself) then the better the estimate of records (and perhaps average record size) the better it can do the dynamic allocation.
Joined: 28 Aug 2007 Posts: 1742 Location: Tirupur, India
Hi Bill,
Thanks for your thought on this. I was partially having the same idea as you did. I was at a dilemma since the saslog says "memory",
Code:
ERROR: THE SAS SYSTEM STOPPED PROCESSING THIS STEP BECAUSE OF INSUFFICIENT MEMORY
and i have seen sort jobs that have below messages, but complete successfully.
Code:
+ICE097I 1 JOBNAME1.S1 SORT RECOVERING FROM B37 ABEND ON SORTWK DATA SET
IEC030I B37-04,IFG0554A,JOBNAME1,SAS,SORTWKB2,554C,PUB528,0427041D,XXXXX.YYYYY.RA000.JOBNAME1.R0142402
also increasing the REGION parameter of the step to 0M makes the job run successfully "sometimes".
I was hesitating to experiment anything, since the job runs for like 10 hours and I don't want it to fail at 9th hour cos of my parm change :-)
Joined: 28 Aug 2007 Posts: 1742 Location: Tirupur, India
A sudden flash of thought, not sure how far it is true
Since SORT is capable of using HIPERSPACE, memory objects & work datasets, perhaps SAS referred the error generically as "INSUFFICIENT MEMORY" instead of "INSUFFICIENT DISK"
Joined: 14 Mar 2007 Posts: 8797 Location: Welsh Wales
Hi Vasanthz,
I had a similar problem a couple of years ago, so will give you the solution that worked for me with help from Frank Yaeger.
Only a mere 27 million records though
S400 abends, with memory shortage messages given if the memory (no pun intended) still works correctly.
I found at the time that for some reason SAS would invoke its own SORT routines rather than DFSORT which we have installed here. So the above forced SAS to use DFSORT from the start and ignore the initial usage of the SAS SORT routines.
Can't recall why we came up with NOSORTBLKMODE, but the original thread and responses are on this forum somewhere.
- - - Just read the link posted by Skolusu, so maybe from there.
Hopefully this may be of some use to you. Funny enough I'm back at the very same site where this oproblem arose, and the job is still running just fine using a REGION=64M
Joined: 28 Aug 2007 Posts: 1742 Location: Tirupur, India
Hi Kolusu,
The link you posted seems to be describing the issue at our shop. I would try out the suggestions mentioned over there along with FILSZ,
Thank you. It is very helpful.
Expat, Thanks to you too, we are planning to experiment with NOSORTBLKMODE as you suggested & the same is mentioned on the link.
