Generate Checksum in mainframe

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

Hello,

Hope you are all fine. I have a requirement to generate checksum. I give the same input file to unix and mainframe. My objective is to verify if both the systems (Unix and Mainframe) are producing the same output (I am working on a migration project).

So, instead of comparing the full file every time, i believe generating a checksum and comparing the checksum will be more effective.

I got to know that there is no utility available to do this. should i write my own program? if yes, do we have an algorithm that produces checksum.

Note: My objective is not about scurity of the file, its just to check the correctness.

Please advise.

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

I am currently checking SAS MD5, AMASPZAP. But these doesn't give me what i want.

MD5 and ZAP gives me a the checksum for a particular record or i have a limitation of 200 bytes. But i want the checksum for the whole file.

enrico-sorichetti · Posted: Tue Jun 17, 2014 6:02 pm

You might want to look at FILE 900 at Use [URL] BBCode for External Links

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

I am currently checking SAS MD5, AMASPZAP. But these doesn't give me what i want.

MD5 and ZAP gives me a the checksum for a particular record or i have a limitation of 200 bytes. But i want the checksum for the whole file.

Ed Goodman · Active Member Joined: 08 Jun 2011 Posts: 556 Location: USA

Do you have the Unix stuff available on the mainframe? Like OMVS?

I used that once to generate a checksum. The problem is that you have to convert the mainframe file to ascii to get the same results.

steve-myers · Posted: Tue Jun 17, 2014 6:41 pm

There are two fundamental problems.

Text data in a mainframe is in the EBCDIC character set; text data on a work station is usually in a different character set.
Data set formats on a mainframe are usually in a different format than a workstation.

So, obtaining a checksum of the data is comparing apples to oranges; you will usually get a different checksum.

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

If not checksum, what is the way to achieve it?

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

I am thinking of writing a record level checksum in another file using MD5 - SAS, and add all the record level checksum to make it one, do the same in unix and compare both.

Will this work? does anybody know?

SAS - MD5 produces checksum for any given string. This is not character set dependent i believe.

Robert Sample · Posted: Tue Jun 17, 2014 8:08 pm

From the SAS documentation:

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

Robert that was very useful. Thanks.

But do you guys agree with the below statements,

1) There is no existing IBM utility that does this.
2) We do not have a sas function that creates a file level checksum.

One more thing,

If you know of any 3rd product that does this, please suggest. My customer is ok to buy a lisence.

Bill Woodger · Posted: Tue Jun 17, 2014 9:15 pm

You won't get any type of MD5 checksum to match across one system using EBCDIC and another using ASCII and one system being "Big Endian" and other being "Little Endian".

How many files do you want to do this for? Is the data entirely "textual" (no binary fields, no floating-point, no packed-decilogicallymal, no BCD, no control-characters (other than for line/file end on the *nix)?

If so, you can do it yourself. You have a table where each textual character is assigned a unique numeric value. You use the same table on the Mainframe and on the other system. You look at each record, byte-by-byte, in the table, and accumulate a total across the entire file.

Note some problems. You could mix-up the data on one file, partially or entirely, and still get the same result. Different data will "likely" stand out (can be made more likely by using higher prime numbers for your reference table), but unless you also add "weight" for character-position, you're a little exposed. You also need to "weight" for record-number (else two records could be swapped and not noticed). Getting trickier, eh?

The really simple way, if the files are supposed to be identical, is to transfer one or the other, so they are both on the same character-set. Then compare. If insisting on checksumming, do some weighting yourself for record position.

I suspect whoever has made the decision to checksum is not aware that it is not really going to work.

And that is with textual-only data...

And that is if the files are actually supposed to be logically identical as well.

Phsyically identical on the two different systems, forget it. Physically identical if both files on the same OS? Possible for textual-date only.

Unless you have a tool which checksums a file (that being its task, so it knows the importance of the order of things, one a record, and within a file) and the data is textual, just forget the checksum as a "checksum".

And please, this time, don't wander off leaving us not knowing the outcome.

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

At this point i have two feasible solutions,

1) Creating a table with unique value for each character and use it in UNIX and MF, write my own checksum routine.

2) Use MD5 SAS. (MF file needs to be converted into ASCII)

But I have around 3000 files to compare and these files will be created every month.

Both the solutions are time consuming, this will also run for a long time, consumes lot of MIPS. So both are costly solutions and i cannot affort this. Still searching for other ways.

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

One more update: I didn't find what i want in Use [URL] BBCode for External Links.

enrico-sorichetti · Posted: Thu Jun 19, 2014 1:00 pm

Ed Goodman · Active Member Joined: 08 Jun 2011 Posts: 556 Location: USA

I built a CRC32 hasher in COBOL, to be able to make a zip-compatible file.

Is that anything at all like what you need?

shankarm · Active User Joined: 17 May 2010 Posts: 175 Location: India

Just wanted to close this. Below is what we did,

We we transferred the file from mainframe to HDFS and wrote a checksum algorithm in hadoop to compare the files generated in both the environment.

Ed Goodman · Active Member Joined: 08 Jun 2011 Posts: 556 Location: USA

nice!

ApexNC · New User Joined: 10 Feb 2006 Posts: 19 Location: USA

publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/DZ9ZR003/7.5.23 - The CHECKSUM instruction is highly precise, but must be coded in Assembler.