IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

Sort card to find duplicates, without changing the record


IBM Mainframe Forums -> DFSORT/ICETOOL
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
anand1204

New User


Joined: 16 Jan 2015
Posts: 12
Location: USA

PostPosted: Fri Aug 31, 2018 12:13 am
Reply with quote

Hi,

I'm looking for a sort control statements (not JCL), to remove duplicates from PS data set (not file) which has an LRECL of [is] 200 bytes and my key is the entire LRECL [file]. But my record order shouldn't change, also the duplicates check should be for particular date only (20180825)

I/P data set (not file)
--------------


Code:
aaa 20180825 dddd.......ccccc
ccc 20180826 aaaa...... ddddd
aaa 20180825 dddd.......ccccc
ggg 20180227 hhhh.......hhhhh
ccc 20180826 aaaa...... aaaaa
aaa 20180825 dddd.......ccccc
ccc 20180826 aaaa...... aaaaa



O/p data set (not file)
----------------

Code:
aaa 20180825 dddd.......ccccc
ccc 20180826 aaaa...... ddddd
ggg 20180227 hhhh.......hhhhh
ccc 20180826 aaaa...... aaaaa
ccc 20180826 aaaa...... aaaaa


Thanks in advance.
Amended to show correct terminology and use code tags. Stuff in [] is to be ignored.
Back to top
View user's profile Send private message
dneufarth

Active User


Joined: 27 Apr 2005
Posts: 418
Location: Inside the SPEW (Southwest Ohio, USA)

PostPosted: Fri Aug 31, 2018 1:28 am
Reply with quote

Key is entire record?


Been a few years for me, but read about adding SEQNUM at end of record as well as SUM FIELDS=NONE when sorting


Probably a two step job with the second step putting records back in order of original file and truncating SEQNUM field. (or one step in ICETOOL)
Back to top
View user's profile Send private message
anand1204

New User


Joined: 16 Jan 2015
Posts: 12
Location: USA

PostPosted: Fri Aug 31, 2018 2:08 am
Reply with quote

dneufarth wrote:
Key is entire record?


Been a few years for me, but read about adding SEQNUM at end of record as well as SUM FIELDS=NONE when sorting


Probably a two step job with the second step putting records back in order of original file and truncating SEQNUM field. (or one step in ICETOOL)


hi,
Thanks for the reply.

Do you have a sample JCL to accomplish this
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2481
Location: @my desk

PostPosted: Fri Aug 31, 2018 3:50 am
Reply with quote

Quote:

duplicates check should be for particular date only (20180825)
where do you get that date from?
Back to top
View user's profile Send private message
anand1204

New User


Joined: 16 Jan 2015
Posts: 12
Location: USA

PostPosted: Fri Aug 31, 2018 4:06 am
Reply with quote

Arun Raj wrote:
Quote:

duplicates check should be for particular date only (20180825)
where do you get that date from?


Hi Arun,

Yes, i'm trying check duplicates only one particular date (20180825) and also i dont need the order sequence to change.

I have also gone thru the previous post from this link below. the last post (solution does change the record sequence)

ibmmainframes.com/about32365.html
Back to top
View user's profile Send private message
dneufarth

Active User


Joined: 27 Apr 2005
Posts: 418
Location: Inside the SPEW (Southwest Ohio, USA)

PostPosted: Fri Aug 31, 2018 5:27 am
Reply with quote

Don't have JCL example. Most responders here prefer you try something after searching for similar requests/solutions. If there is an issue, then post what you tried and system messages.

Arun and others usually have tremendous knowledge and some tricks up their sleeves (time permitting).
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3048
Location: NYC,USA

PostPosted: Fri Aug 31, 2018 6:04 am
Reply with quote

As said , please do spend time to research yourself since it’s so common question.
Look here for similar problem and you all need do it twist little bit.
www.ibmmainframeforum.com/dfsort-icetool-icegener/topic4480.html
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2481
Location: @my desk

PostPosted: Fri Aug 31, 2018 10:41 am
Reply with quote

Quote:
Yes, i'm trying check duplicates only one particular date (20180825) and also i dont need the order sequence to change.
anand1204,

Where do you get this 'one particular date' from?, I am just trying to make sure I fully understand what you are trying to do
Back to top
View user's profile Send private message
sergeyken

Senior Member


Joined: 29 Apr 2008
Posts: 2010
Location: USA

PostPosted: Fri Aug 31, 2018 6:08 pm
Reply with quote

anand1204 wrote:
i'm trying check duplicates only one particular date (20180825) and also i dont need the order sequence to change.

I have also gone thru the previous post from this link below. the last post (solution does change the record sequence)

ibmmainframes.com/about32365.html

Your comments demonstrated that you do not try to understand the recommended algorithm, but just tried to copy-paste the given example as is.
In order to keep your output records in their original order you need to add one extra step at the end (by yourself!), to re-sort them by SEQNUM field created in that example, and probably remove this field at the very end.
I intentionally do not give the ready to copy-paste solutions to those who doesn't try using his own mind to do his own job.
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2481
Location: @my desk

PostPosted: Fri Aug 31, 2018 8:51 pm
Reply with quote

anand1204,

Also in your sample data all the duplicate records are identical for the input date - 20180825. Will it be the case always?, or do you expect different data on these duplicate records?.
Back to top
View user's profile Send private message
Rohit Umarjikar

Global Moderator


Joined: 21 Sep 2010
Posts: 3048
Location: NYC,USA

PostPosted: Fri Aug 31, 2018 8:53 pm
Reply with quote

Arun Raj, OP is already loaded with many questions by us to try it by its own and may refer the link provided. Let us wait for it than spending anymore time with unclear requirements.
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2481
Location: @my desk

PostPosted: Fri Aug 31, 2018 9:22 pm
Reply with quote

Quote:
OP is already loaded with many questions by us to try it by its own
I don't see too many questions asked here, except the one from Dave.
Back to top
View user's profile Send private message
Nic Clouston

Global Moderator


Joined: 10 May 2007
Posts: 2455
Location: Hampshire, UK

PostPosted: Sat Sep 01, 2018 3:59 pm
Reply with quote

Quote:
I don't see too many questions asked here, except the one from Dave.

Plus 3 from you, Arun, one of which should not have been asked as the information (is it only that date?) was provided in the original post. Of the other two I do not see the relevance to providing a solution but I do see the relevance of multiple identical records. Can there be 3 duplicates of which 2 are valid and should be retained? Probably not but OP should be made aware of the ramifications.

As I see it, someone mucked up the run that created records for that date - did not back out a failure or something. But that is irrelevant to the solution of getting the data fixed up THIS TIME. The solution to fixing up the data is as dneufarth posted right at the start: add a sequence number as the data is read in - at the end of each record (bytes201-2xx depending), doing a SORT FIELDS=NONE and saving the data. Then sort again - on the added sequence number, doing a build to remove the sequence number from the output data. Examples of all this can be found in the forum and in the manuals.

Note: RECFM should have been provided.

Perhaps the soution could be made to accept the date as a parameter so that the JCL can be kept and reused if a similar event happens again. But, more importantly, the whole application should be designed so that this cannot happen - no appending to a data set but create new records to a temp data set and concatenate after sucessful execution.

Topic has unlocked itself during my cleanup. I have also edited the original post.
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2481
Location: @my desk

PostPosted: Sat Sep 01, 2018 8:39 pm
Reply with quote

Thanks Nic. You're right. I have corrected it now.

OP mentioned in a PM while the topic was locked that he needs only one record out of the duplicate-date records. If that is the case then probably we don't have to sort the records or re-sort it.

It could be just assigning a sequence number for records with date = '20180825' (using an IFTHEN..WHEN...OVERLAY) and then omitting records having sequence number > 1 in the output (OUTFIL OMIT....).
Back to top
View user's profile Send private message
anand1204

New User


Joined: 16 Jan 2015
Posts: 12
Location: USA

PostPosted: Wed Sep 05, 2018 2:01 am
Reply with quote

Arun Raj wrote:
Thanks Nic. You're right. I have corrected it now.

OP mentioned in a PM while the topic was locked that he needs only one record out of the duplicate-date records. If that is the case then probably we don't have to sort the records or re-sort it.

It could be just assigning a sequence number for records with date = '20180825' (using an IFTHEN..WHEN...OVERLAY) and then omitting records having sequence number > 1 in the output (OUTFIL OMIT....).


Hi Arun,

Thanks for your response & the logic for this scenario.
Back to top
View user's profile Send private message
Arun Raj

Moderator


Joined: 17 Oct 2006
Posts: 2481
Location: @my desk

PostPosted: Wed Sep 05, 2018 2:45 am
Reply with quote

Anand.

Thanks for posting. Glad it worked for you.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> DFSORT/ICETOOL

 


Similar Topics
Topic Forum Replies
No new posts SFTP Issue - destination file record ... All Other Mainframe Topics 2
No new posts FINDREP - Only first record from give... DFSORT/ICETOOL 3
No new posts JCL sort card - get first day and las... JCL & VSAM 9
No new posts To find whether record count are true... DFSORT/ICETOOL 6
No new posts Validating record count of a file is ... DFSORT/ICETOOL 13
Search our Forums:

Back to Top