|
View previous topic :: View next topic
|
| Author |
Message |
jmitchell
New User
Joined: 02 Oct 2020
Posts: 7
Location: United States
|
|
|
|
Good afternoon. I am heading up a project in our department to convert several hundred Easytrieve Plus programs to Python. The advantages are numerous. Python is an interpretive language like Easytrieve, there are no licensing fees, there are a bunch of kids coming out of school with that skill set....on and on.
We have Python 3.8.5 installed in the USS partition of our z/OS IBM mainframe.
My JCL to invoke Python using IBM's BPXBATCH module is:
| Code: |
//xxxxxMH1 JOB (11111,1),'PYTHON ',TIME=(,15),
// NOTIFY=&SYSUID,MSGLEVEL=(1,1),
// REGION=0M,
// CLASS=T
/*ROUTE PRINT Q
//***********************************************
//STEP10 EXEC PGM=BPXBATCH
//STDPARM DD DSN=XXXXXMH.TEST.PYTHON(PYPATH),
// DISP=SHR
// DD DSN=XXXXXMH.TEST.PYTHON(PYVER),
// DISP=SHR
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=* |
In the PYPATH file, the single line invokes Python using the following string:
sh /usr/lpp/IBM/cyp/v3r3/pyz/bin/python3
In the PYVER file, I put in this line:
This runs perfectly in batch. The STDOUT DD produces this 1 line report:
Python 3.8.5
However, if I change the PYVER file to include a hello world statement like this:
| Code: |
--version
print('Hello world') |
I only get the STDERR DD producing this:
| Code: |
| FSUM7332 syntax error: got (, expecting Newline |
I believe the problem is the ASCII/EBCDIC translation when going to the USS partition.
I've tried just about everything I can think of. I've tried setting codepages as recommended by Python, but those require a # to begin the statement, and when I do that, Python can't see any statements after that. It may be theat the hashtag is not translated correctly. However, since the first example, with only the Python --version statement comes back clean, I believe I am hooking into the Python interpreter correctly.
I've tried to get the Systems Programmers to fix it, but they know nothing about it. We have a 30 minute call with IBM in a week, but this is crazy. I can't be the first guy to be doing this.
I find NO information on the Internet about Python's JCL in z/OS, but people are apparently running it in the USS partition.
Any insight would be enormously appreciated.
JMitchell
Please use the code tags when posting code and data - done for you this time |
|
| Back to top |
|
 |
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1329
Location: Bamberg, Germany
|
|
|
|
| While STDPARM is able to accept a --version parameter, I don't think this applies for Python code itself. You could try that via STDIN. |
|
| Back to top |
|
|
jmitchell
New User
Joined: 02 Oct 2020
Posts: 7
Location: United States
|
|
|
|
Joerg,
Thanks for that info. I added the STDIN DD statement in the JCL and created the one line file with the print('Hello world') statement. It died with the the same error. This is good news somewhat. Now I know that I need to put the program in the STDIN statement.
I also tried a line ahead of the print statement that was # coding=IBM-1047. As before, once Python saw the hashtag, it ignored the rest of the code, so the print statement was bypassed.
I do appreciate your input very much.
JMitchell |
|
| Back to top |
|
|
Joerg.Findeisen
Senior Member
Joined: 15 Aug 2015
Posts: 1329
Location: Bamberg, Germany
|
|
|
|
| You might have a look at ibm.com how to set up STDENV for a Locale. May it be of help. |
|
| Back to top |
|
|
WilliamDownie
New User
Joined: 01 Jul 2020
Posts: 21
Location: UK
|
|
|
|
I've been tinkering with running OMVS Python scripts from from z/OS. I've just started looking into this so may not be the correct way of setting things up, but it may help you .
The error you have got looks like print('Hello world') is not being interpreted as Python.
What I've done to run a OMVS Python script from z/OS is as follows:
JCL
| Code: |
//STEP1 EXEC PGM=BPXBATCH
//STDPARM DD DISP=SHR,DSN=Z01408.SCRIPTS(RUNSH)
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
|
Z01408.SCRIPTS(RUNSH)
| Code: |
sh /z/z01408/hello.sh
|
hello.sh
| Code: |
#!/bin/sh
python3 hello.py
|
hello.py This resides in /z/z01408
| Code: |
print("hello world")
|
Output from STDOUT is:
| Code: |
******************* TOP OF DATA ******
hello world
****************** BOTTOM OF DATA ******
|
Note, if I change "Z01408.TEST.PYTHON(RUNSH)" to "sh /z/z01408/hello.py" I get this error:
| Code: |
/z/z01408/hello.py 1: FSUM7332 syntax error: got (, expecting Newline
|
which is why I think , in your example, your script is not being interpreted as Python. |
|
| Back to top |
|
|
Apoorva
New User
Joined: 28 Jan 2020
Posts: 49
Location: India
|
|
|
|
| WilliamDownie wrote: |
which is why I think , in your example, your script is not being interpreted as Python. |
Informative. Thanks for the info! |
|
| Back to top |
|
|
enrico-sorichetti
Superior Member
Joined: 14 Mar 2007
Posts: 10886
Location: italy
|
|
|
|
| Quote: |
| sh /z/z01408/hello.py |
You just told sh to interpret a python script
sh in this case disregards the shebang line
it is like expecting the cobol compiler to understand a fortran source |
|
| Back to top |
|
|
jmitchell
New User
Joined: 02 Oct 2020
Posts: 7
Location: United States
|
|
|
|
@Joerg.Findeisen, @WilliamDownie, @Apoorva, @enrico-sorichetti
You are all Rock Stars!!!!!!
Using information from you all, I was able to get my first Python program running from a batch environment on the mainframe. Below is the JCL and members that I used:
| Code: |
//xxxxxMHP JOB (#####,1),’PYTHON’,TIME=(,15),
// NOTIFY=&SYSUID,MSGLEVEL=(1,1),
// REGION=0M,
// CLASS=T
/*ROUTE PRINT Q
//*****************
//STEP10 EXEC PGM=BPXBATCH
//STDPARM DD DSN=xxxxxMH.TEST.SCRIPTS(RUNSH),
// DISP=SHR
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=* |
In my xxxxxMH.TEST.SCRIPTS(RUNSH) file, there is a single line where IBM recommends the installation of Python, which our System Programmers followed, followed by a space, then the Python source code name:
sh /usr/lpp/IBM/cyp/v3r8/pyz/bin/python3 /tmp/hello.py
In the ISPF 3.17 menu, I access the z/OS UNIX Directory List. I created the hello.py member in the /SYSTEM/tmp directory. This is the file:
| Code: |
print('Hello world')
a = 4
b = 6
c = a+b
print('a = '+str(a))
print('b = '+str(b))
print('a + b = '+str(c)) |
I was delighted to see this simple output:
Hello world
| Code: |
a = 4
b = 6
a + b = 10
|
It looks pretty simple when you see it, but I ran hundreds of iterations of trials and errors to get to this point.
My next challenge is to be able to assign the source code to a dataset that is in the STDIN DD statement for the invoking batch program BPXBATCH. If anyone has suggestions, I am very grateful.
I really do appreciate you all taking your time to help me all. Collaboration across the globe!
JMitchell |
|
| Back to top |
|
|
Apoorva
New User
Joined: 28 Jan 2020
Posts: 49
Location: India
|
|
|
|
| jmitchell wrote: |
@Joerg.Findeisen, @WilliamDownie, @Apoorva, @enrico-sorichetti
You are all Rock Stars!!!!!!
JMitchell |
Glad to know that it worked! I am no "Rock Star" because I didn't provide any inputs  . I was carefully following this post because I have also been running a Python application (on distributed server) that integrates JIRA with the Mainframes (basically captures error messages in z/os, and logs it in JIRA as issues). If I could run my python application on z/os (like you did) I could probably deploy that python application on z/os itself. |
|
| Back to top |
|
|
WilliamDownie
New User
Joined: 01 Jul 2020
Posts: 21
Location: UK
|
|
|
|
| Quote: |
| My next challenge is to be able to assign the source code to a dataset that is in the STDIN DD statement for the invoking batch program BPXBATCH |
The following doesn't assign the source code to STDIN, so maybe not what you are after, but as I spent some time looking into it, I thought I'd post incase it gives you some ideas.
According to the documentation I've found BPXBATCH can run a shell script or an executable program. In your case you are running a shell script to invoke the Python script. The manual says "... use STDIN to define the name of the shell script to be invoked."
You could point to the script in STDIN:
| Code: |
//STEP1 EXEC PGM=BPXBATCH
//STDIN DD PATH='/z/z01408/directory1/hello',PATHOPTS=(ORDONLY)
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
|
where , in my case, hello is:
| Code: |
| python3 /z/z01408/hello.py |
and hello.py is
| Code: |
| print("hello world") |
Running this output the "hello world" to STDOUT.
Note, in the above I did not need to specify this was a shell script (using "sh") because that is the default.
It look me a while to find the IBM manual (I couldn't find it on the IBM web site). So, if you'd like me to forward it to you let me know. I find the manual easier to use that IBM's online resource (as I find clicking from page to page annoying). |
|
| Back to top |
|
|
jmitchell
New User
Joined: 02 Oct 2020
Posts: 7
Location: United States
|
|
|
|
First, I want to apologize to @enrico-sorichetti for not including you in the Rock Star category. You really are. I was under the impression the --version statement was interpreted by Python. Then I ran it in Juptyter and it didn't recognize it as a Python command. That led me to not using this as a verification of the connection to Python.
So, my dilemma is that my organization needs to run batch programs from an IBM z/OS MVS environment that used to use Easytrieve, and wants to switch to Python. We run these jobs using JCL. There is no Python program that IBM has installed in the MVS environment. They install it in the z/OS Unix environment, which is a separate LPAR located next to the MVS LPAR.
Using IBM utility program BPXBATCH, I can set a parameter to activate Python in the next door Unix environment, and pass in the name of the Python program I want it to execute. That program must exist (as far as I know) inside the Unix world. The reports are returned to the MVS world thru the DD STDOUT.
The problem is, I need the program to be able to live in the MVS environment as a dataset member, so it conforms to our traditional change control management requirements. It also must be able to read unlimited various files and formats from many DD statements, as well as write various unlimited files. In our environment, our programs may read/write 100-500 million records here or there. This is no Mom and Pop shop.
I have been able to create a Python stub program in the Unix LPAR that opens and executes another file containing the desired Python program. However, that called Python program is also existing in the Unix LPAR. I need to be able to call a specific DD statement in the MVS JCL that will contain the path to the program. For Easytrieve, it was the SYSIN DD.
I cannot even figure out how to open any MVS DD files from the Unix LPAR. I have found that in Python 2.7 there was a ddreader.py, but I can't find any further references than that.
I'm pretty exhausted with how difficult this is. I would think IBM would have supplied a regular program, so in your JCL you could specify PGM=PYTHON and have access to all the IBM file services.
Any encouragement, hope, or insight is appreciated.
JMitchell |
|
| Back to top |
|
|
enrico-sorichetti
Superior Member
Joined: 14 Mar 2007
Posts: 10886
Location: italy
|
|
| Back to top |
|
|
WilliamDownie
New User
Joined: 01 Jul 2020
Posts: 21
Location: UK
|
|
|
|
| Quote: |
| I cannot even figure out how to open any MVS DD files from the Unix LPAR |
Have you looked into the Python library for Z Open Automation Utilities (ZOAU) ? I found out about this through doing this years "Master the Mainframe". For details do a search on "zoautil-python". But I don't know whether ZOWE needs to be installed to use these (ZOWE formed a large part of MTM2020). I wrote a Python script to get details on job output (using zoautil_py.Jobs). I can send this onto you if you'd like an example of how to use the ZOAU library.
I found a post on Stack Overflow which has an example of a Python script that wraps USS commands (I didn't try it), which would be another way of accessing MVS datasets when running in OMVS.
See : stackoverflow.com/questions/59973962/utf-8-to-ebcdic-using-iconv-in-python-script-on-uss
With these two techniques you will have to specify a dataset name rather than referencing a JCL DD name.
If you need to keep the Python source code in MVS, perhaps you could have a step that copies the script to a USS directory before running it. |
|
| Back to top |
|
|
enrico-sorichetti
Superior Member
Joined: 14 Mar 2007
Posts: 10886
Location: italy
|
|
|
|
| Quote: |
| If you need to keep the Python source code in MVS, perhaps you could have a step that copies the script to a USS directory before running it. |
that' s what the script I was talking about does
I know that the forum frowns on the attachments, but since they are PDFS, You will have to live with that 
since it looks like that there are not any IP issues here they are |
|
| Back to top |
|
|
jmitchell
New User
Joined: 02 Oct 2020
Posts: 7
Location: United States
|
|
|
|
Hello everyone,
We had our meeting with IBM today. It was a short introduction of the players, and to define what our issues are trying to replace Easytrieve in the MVS environment with Python.
They say they have been working with a few organizations on this. They only really installed Python in June 2020 in the z/OS environment from what I understood from the meeting, so this is really cutting edge for this environment.
Next week, we will be getting into more of the nuts and bolts, and I hope that I can provide everyone with more information about how this stuff works. So far, I can execute a Python program in the Unix side from JCL in the MVS side. I can open and read a Unix file and create an MVS dataset with the copied data. I can also do a LISTC command to print all the attributes to an MVS dataset. However, the SUPER frustrating issue is that I cannot figure out how to open that dataset to read it!!!!! I've spent HOURS trying various things. I have used all the methods you all have provided, which actually got me this close.
Anyways, I will continue to keep you updated. This thread will undoubtedly be searched by other developers in the future as more and more companies with z/OS systems try and shift to Python.
Thank you all,
JMitchell |
|
| Back to top |
|
|
Hungryhead
New User
Joined: 21 Apr 2022
Posts: 1
Location: India
|
|
|
|
Hi Mitchell,
I am trying similar thing for our application.
Have you got further learning or updates from your IBM team?
I would love to hear it and explore it. |
|
| Back to top |
|
|
jmitchell
New User
Joined: 02 Oct 2020
Posts: 7
Location: United States
|
|
|
|
Hungryhead,
I'm sad to report that I wasn't able to crack that nut. Other project priorities took away the time to continue this research. If you finally triumph over this challenge, please share.
Thanks, and good luck!
Mitch |
|
| Back to top |
|
|
RonnyFunk
New User
Joined: 25 Apr 2022
Posts: 2
Location: Germany
|
|
|
|
Hi,
just in case your dataset read problem has not been solved already. You actually need to use the python package provided with the IBM Z Open Automation Utilities. Below is a quick and dirty example of how to do that.
In case you need to install the utilities you can find the doc here and the Python api reference is here
greetings
Ronny
| Code: |
#######################################################################
# Example IBM Enterprise Python and IBM Z Open Automation Utlities #
#######################################################################
import sys,os
import time
import argparse
from zoautil_py import datasets, mvscmd
from zoautil_py.types import DDStatement, DatasetDefinition, FileDefinition
################################################################################
# Parse passed arguments #
################################################################################
def parse_our_args(argv=None):
program_name = os.path.basename(sys.argv[0])
if argv is None:
argv = sys.argv[1:]
try:
parser = argparse.ArgumentParser(program_name)
parser.add_argument("-dsn", "--dataset-name", help="Dataset Name", required=True)
opts = parser.parse_args(argv)
return opts
except Exception as e:
indent = len(program_name) * " "
sys.stderr.write(f'{program_name}:{repr(e)}\n')
sys.stderr.write(f'{indent} for help use --help\n')
sys.exit(16)
def main(dsn):
if datasets.exists(dsn):
print(datasets.read(dsn))
rc = 0
else:
print(f'Dataset {dsn} not found')
rc = 12
return rc
if __name__ == '__main__':
args = parse_our_args()
retcode = main(args.dataset_name)
sys.exit(retcode)
|
|
|
| Back to top |
|
|
hankoerlemans
New User
Joined: 25 Jan 2018
Posts: 61
Location: Australia
|
|
|
|
Umm never saw this post so I went with (some time ago) IEBGENER to output an INSTREAM python program into a temp sequential file.
Second IEBGENER to output a Rexx program to a temp PDS for a TSO step.
I find most OMVS stuff easiest to do with Rexx BTW.
So one of things the Rexx does is
| Code: |
/* This is easier than trying to work out how to get
newlines added via IEBGENER or some such */
Address TSO "execio * diskr msglist (stem lines. finis"
'writefile msglist.py 700 lines. 0'
|
After the file exists you can "chtag -tc 1047 msglist.py;" to make sure it's interpreted properly. I happened to do that with a spawn of /bin/sh .
Elsewhere I invoke said msglist.py with
| Code: |
/* Python to parse the list */
"pipe so."
"pipe se."
map.1 = so.2
map.2 = se.2
arg.3 = 'python3 msglist.py'
"spawnp (arg.1) 3 map. arg. env."
pid = retval
if pid < 1 then do; Say 'Spawn failed for python3';exit 99;end;
"wait wst."
"close "so.2
"close "se.2
say 'STDOUT:'
fd = so.1
call pythonout
say 'STDERR: 'buf
fd = se.1
call pythonout
"close "so.1
"close "se.1
if wst.w_exitstatus > 0 then do
say 'STDERR: python3 ended with RC='wst.w_exitstatus
exit 99
end
|
So execution really starts with a batch TSO step invoking the Rexx which drives OMVS shell and python.
Possibly a bit convoluted but it helped with my investigations and prototyping the python code quickly. i.e. ISPF edit is quick for me :-)
Once stable it can be stored somewhere useful. |
|
| Back to top |
|
|
RonnyFunk
New User
Joined: 25 Apr 2022
Posts: 2
Location: Germany
|
|
|
|
HungryHead,
This is not only a bit convoluted it basically is in the back, through the chest in the eye  The problem is, IBM does not promote IBM Z Open Automation Utilities etc. enough. Every z Shop should know about them, where to find the documentation and how to use them with Shell Scripts and with IBM Enterprise SDK for Python.As an example here is a PDS Compress in Place from inside z/Unix (USS) Shell (bash in this case). SYSIN comes from stdin, SYSPRINT goes to stdout:
| Code: |
| echo " COPY OUTDD=DD2,INDDD=DD1" | mvscmdauth --pgm=iebcopy --DD1=MY.PDS,SHR --DD2=MY.PDS,SHR --SYSIN=stdin –SYSPRINT=* |
The RC from IEBCOPY is available inside the Shell $?
The utilities are one piece of the software required to make z/OS way more modern. They are hiding everything behind wired terms like IBM Wazi Developer for Code Ready Work Spaces. But every z Shop can modernize their z/Unix environment today and could have started at least two years ago. It does not require Openshift or the like. Basically everything you need is already part of z/OS or can be downloaded directly using your IBM Id or from IBM Shop Z as pax or SMP/E installs.
Here is what you need to do:
z/OS
1) Install ported software from Rocket using miniconda (bash, git , make etc)
2) Install IBM Enterprise SDK for Python and IBM Enterprise SDK for Go and IBM Enterprise SDK for Nodejs
3) Install IBM Z Open Automation Utilities
4) Customize your /etc/profile (very important)
5) Customize z/OSMF REST API
6) Customize z/OS NFS Server
7) If your shop does fullfill the requirements for z/OS Container Extentions (z/CX) use z/OSMF Workflows to customize it
Desktop Software
1) Install Nodejs
2) Install MS VS Code and use it to install the following VS Code Extensions:
IBM Z Open Editor
Zowe Explorer for MVS
3) Install ZOWE CLI and at least the Secure Credential Store for ZOWE CLI
4) Install MS Windows Terminal
With this in place you will have a modern development experience attractive to young developers and old but young minded sysprogs (as myself) .
greetings
Ronny |
|
| Back to top |
|
|
hankoerlemans
New User
Joined: 25 Jan 2018
Posts: 61
Location: Australia
|
|
|
|
No doubt you are correct.
I will defer to you on this but I never had much time for the z/OSMF monolith.
I'm happy with my nuts and bolts for what I need. |
|
| Back to top |
|
|
|
|
