IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

JCL to invoke Python in z/OS


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 6
Location: United States

PostPosted: Tue Dec 08, 2020 4:05 am
Reply with quote

Good afternoon. I am heading up a project in our department to convert several hundred Easytrieve Plus programs to Python. The advantages are numerous. Python is an interpretive language like Easytrieve, there are no licensing fees, there are a bunch of kids coming out of school with that skill set....on and on.

We have Python 3.8.5 installed in the USS partition of our z/OS IBM mainframe.

My JCL to invoke Python using IBM's BPXBATCH module is:

Code:
//xxxxxMH1 JOB (11111,1),'PYTHON ',TIME=(,15),
//         NOTIFY=&SYSUID,MSGLEVEL=(1,1),       
//         REGION=0M,                           
//         CLASS=T                               
/*ROUTE    PRINT Q                               
//***********************************************
//STEP10   EXEC PGM=BPXBATCH                     
//STDPARM  DD DSN=XXXXXMH.TEST.PYTHON(PYPATH),   
//            DISP=SHR                           
//         DD DSN=XXXXXMH.TEST.PYTHON(PYVER),   
//            DISP=SHR                           
//STDOUT   DD SYSOUT=*                           
//STDERR   DD SYSOUT=*


In the PYPATH file, the single line invokes Python using the following string:
sh /usr/lpp/IBM/cyp/v3r3/pyz/bin/python3

In the PYVER file, I put in this line:
Code:
--version


This runs perfectly in batch. The STDOUT DD produces this 1 line report:
Python 3.8.5

However, if I change the PYVER file to include a hello world statement like this:
Code:
--version
print('Hello world')


I only get the STDERR DD producing this:
Code:
FSUM7332 syntax error: got (, expecting Newline


I believe the problem is the ASCII/EBCDIC translation when going to the USS partition.

I've tried just about everything I can think of. I've tried setting codepages as recommended by Python, but those require a # to begin the statement, and when I do that, Python can't see any statements after that. It may be theat the hashtag is not translated correctly. However, since the first example, with only the Python --version statement comes back clean, I believe I am hooking into the Python interpreter correctly.

I've tried to get the Systems Programmers to fix it, but they know nothing about it. We have a 30 minute call with IBM in a week, but this is crazy. I can't be the first guy to be doing this.

I find NO information on the Internet about Python's JCL in z/OS, but people are apparently running it in the USS partition.

Any insight would be enormously appreciated.

JMitchell

Please use the code tags when posting code and data - done for you this time
Back to top
View user's profile Send private message
Joerg.Findeisen

Active User


Joined: 15 Aug 2015
Posts: 396
Location: Bamberg, Germany

PostPosted: Tue Dec 08, 2020 4:28 am
Reply with quote

While STDPARM is able to accept a --version parameter, I don't think this applies for Python code itself. You could try that via STDIN.
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 6
Location: United States

PostPosted: Tue Dec 08, 2020 5:32 am
Reply with quote

Joerg,

Thanks for that info. I added the STDIN DD statement in the JCL and created the one line file with the print('Hello world') statement. It died with the the same error. This is good news somewhat. Now I know that I need to put the program in the STDIN statement.

I also tried a line ahead of the print statement that was # coding=IBM-1047. As before, once Python saw the hashtag, it ignored the rest of the code, so the print statement was bypassed.

I do appreciate your input very much.

JMitchell
Back to top
View user's profile Send private message
Joerg.Findeisen

Active User


Joined: 15 Aug 2015
Posts: 396
Location: Bamberg, Germany

PostPosted: Tue Dec 08, 2020 1:01 pm
Reply with quote

You might have a look at ibm.com how to set up STDENV for a Locale. May it be of help.
Back to top
View user's profile Send private message
WilliamDownie

New User


Joined: 01 Jul 2020
Posts: 15
Location: UK

PostPosted: Tue Dec 08, 2020 2:12 pm
Reply with quote

I've been tinkering with running OMVS Python scripts from from z/OS. I've just started looking into this so may not be the correct way of setting things up, but it may help you .

The error you have got looks like print('Hello world') is not being interpreted as Python.

What I've done to run a OMVS Python script from z/OS is as follows:

JCL
Code:
                                                         
//STEP1    EXEC PGM=BPXBATCH                                     
//STDPARM  DD   DISP=SHR,DSN=Z01408.SCRIPTS(RUNSH)               
//STDOUT   DD   SYSOUT=*                                         
//STDERR   DD   SYSOUT=*                                                                                                                                                                                                       

Z01408.SCRIPTS(RUNSH)
Code:

sh /z/z01408/hello.sh

hello.sh
Code:

#!/bin/sh
python3 hello.py


hello.py This resides in /z/z01408
Code:

print("hello world")

Output from STDOUT is:
Code:

******************* TOP OF DATA ******
hello world 
****************** BOTTOM OF DATA ******


Note, if I change "Z01408.TEST.PYTHON(RUNSH)" to "sh /z/z01408/hello.py" I get this error:

Code:

/z/z01408/hello.py 1: FSUM7332 syntax error: got (, expecting Newline 


which is why I think , in your example, your script is not being interpreted as Python.
Back to top
View user's profile Send private message
Apoorva

New User


Joined: 28 Jan 2020
Posts: 49
Location: India

PostPosted: Tue Dec 08, 2020 4:03 pm
Reply with quote

WilliamDownie wrote:

which is why I think , in your example, your script is not being interpreted as Python.


Informative. Thanks for the info!
Back to top
View user's profile Send private message
enrico-sorichetti

Senior Moderator


Joined: 14 Mar 2007
Posts: 10727
Location: italy

PostPosted: Tue Dec 08, 2020 4:41 pm
Reply with quote

Quote:
sh /z/z01408/hello.py


You just told sh to interpret a python script

sh in this case disregards the shebang line

it is like expecting the cobol compiler to understand a fortran source
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 6
Location: United States

PostPosted: Wed Dec 09, 2020 6:17 am
Reply with quote

@Joerg.Findeisen, @WilliamDownie, @Apoorva, @enrico-sorichetti

You are all Rock Stars!!!!!!

Using information from you all, I was able to get my first Python program running from a batch environment on the mainframe. Below is the JCL and members that I used:

Code:
//xxxxxMHP JOB (#####,1),’PYTHON’,TIME=(,15),
//         NOTIFY=&SYSUID,MSGLEVEL=(1,1),       
//         REGION=0M,                           
//         CLASS=T                               
/*ROUTE    PRINT Q                               
//*****************
//STEP10   EXEC PGM=BPXBATCH                     
//STDPARM  DD DSN=xxxxxMH.TEST.SCRIPTS(RUNSH), 
//            DISP=SHR                           
//STDOUT   DD SYSOUT=*                           
//STDERR   DD SYSOUT=*


In my xxxxxMH.TEST.SCRIPTS(RUNSH) file, there is a single line where IBM recommends the installation of Python, which our System Programmers followed, followed by a space, then the Python source code name:
sh /usr/lpp/IBM/cyp/v3r8/pyz/bin/python3 /tmp/hello.py


In the ISPF 3.17 menu, I access the z/OS UNIX Directory List. I created the hello.py member in the /SYSTEM/tmp directory. This is the file:

Code:
print('Hello world')   
a = 4                   
b = 6                   
c = a+b                 
print('a = '+str(a))   
print('b = '+str(b))   
print('a + b = '+str(c))     


I was delighted to see this simple output:
Hello world
Code:
a = 4         
b = 6         
a + b = 10   

It looks pretty simple when you see it, but I ran hundreds of iterations of trials and errors to get to this point.

My next challenge is to be able to assign the source code to a dataset that is in the STDIN DD statement for the invoking batch program BPXBATCH. If anyone has suggestions, I am very grateful.

I really do appreciate you all taking your time to help me all. Collaboration across the globe!

JMitchell
Back to top
View user's profile Send private message
Apoorva

New User


Joined: 28 Jan 2020
Posts: 49
Location: India

PostPosted: Wed Dec 09, 2020 11:28 am
Reply with quote

jmitchell wrote:
@Joerg.Findeisen, @WilliamDownie, @Apoorva, @enrico-sorichetti

You are all Rock Stars!!!!!!
JMitchell


Glad to know that it worked! I am no "Rock Star" because I didn't provide any inputs icon_smile.gif. I was carefully following this post because I have also been running a Python application (on distributed server) that integrates JIRA with the Mainframes (basically captures error messages in z/os, and logs it in JIRA as issues). If I could run my python application on z/os (like you did) I could probably deploy that python application on z/os itself.
Back to top
View user's profile Send private message
WilliamDownie

New User


Joined: 01 Jul 2020
Posts: 15
Location: UK

PostPosted: Wed Dec 09, 2020 6:47 pm
Reply with quote

Quote:
My next challenge is to be able to assign the source code to a dataset that is in the STDIN DD statement for the invoking batch program BPXBATCH

The following doesn't assign the source code to STDIN, so maybe not what you are after, but as I spent some time looking into it, I thought I'd post incase it gives you some ideas.

According to the documentation I've found BPXBATCH can run a shell script or an executable program. In your case you are running a shell script to invoke the Python script. The manual says "... use STDIN to define the name of the shell script to be invoked."

You could point to the script in STDIN:

Code:

//STEP1    EXEC PGM=BPXBATCH                                         
//STDIN    DD   PATH='/z/z01408/directory1/hello',PATHOPTS=(ORDONLY) 
//STDOUT   DD   SYSOUT=*                                             
//STDERR   DD   SYSOUT=*                                             


where , in my case, hello is:
Code:
python3 /z/z01408/hello.py

and hello.py is
Code:
print("hello world")

Running this output the "hello world" to STDOUT.

Note, in the above I did not need to specify this was a shell script (using "sh") because that is the default.

It look me a while to find the IBM manual (I couldn't find it on the IBM web site). So, if you'd like me to forward it to you let me know. I find the manual easier to use that IBM's online resource (as I find clicking from page to page annoying).
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 6
Location: United States

PostPosted: Thu Dec 10, 2020 6:51 am
Reply with quote

First, I want to apologize to @enrico-sorichetti for not including you in the Rock Star category. You really are. I was under the impression the --version statement was interpreted by Python. Then I ran it in Juptyter and it didn't recognize it as a Python command. That led me to not using this as a verification of the connection to Python.

So, my dilemma is that my organization needs to run batch programs from an IBM z/OS MVS environment that used to use Easytrieve, and wants to switch to Python. We run these jobs using JCL. There is no Python program that IBM has installed in the MVS environment. They install it in the z/OS Unix environment, which is a separate LPAR located next to the MVS LPAR.

Using IBM utility program BPXBATCH, I can set a parameter to activate Python in the next door Unix environment, and pass in the name of the Python program I want it to execute. That program must exist (as far as I know) inside the Unix world. The reports are returned to the MVS world thru the DD STDOUT.

The problem is, I need the program to be able to live in the MVS environment as a dataset member, so it conforms to our traditional change control management requirements. It also must be able to read unlimited various files and formats from many DD statements, as well as write various unlimited files. In our environment, our programs may read/write 100-500 million records here or there. This is no Mom and Pop shop.

I have been able to create a Python stub program in the Unix LPAR that opens and executes another file containing the desired Python program. However, that called Python program is also existing in the Unix LPAR. I need to be able to call a specific DD statement in the MVS JCL that will contain the path to the program. For Easytrieve, it was the SYSIN DD.

I cannot even figure out how to open any MVS DD files from the Unix LPAR. I have found that in Python 2.7 there was a ddreader.py, but I can't find any further references than that.

I'm pretty exhausted with how difficult this is. I would think IBM would have supplied a regular program, so in your JCL you could specify PGM=PYTHON and have access to all the IBM file services.

Any encouragement, hope, or insight is appreciated.

JMitchell
Back to top
View user's profile Send private message
enrico-sorichetti

Senior Moderator


Joined: 14 Mar 2007
Posts: 10727
Location: italy

PostPosted: Thu Dec 10, 2020 2:34 pm
Reply with quote

You have nothing to apologise about icon_biggrin.gif

googling around I found this link

www.linkedin.com/pulse/running-unix-python-perl-php-scripts-via-jcl-zos-jimmy-spijker?trk=public_profile_article_view

the script and the jcl are unloadable as PDFs

even if the script is pretty complicated , it is worth exploring

good luck
Back to top
View user's profile Send private message
WilliamDownie

New User


Joined: 01 Jul 2020
Posts: 15
Location: UK

PostPosted: Thu Dec 10, 2020 4:00 pm
Reply with quote

Quote:
I cannot even figure out how to open any MVS DD files from the Unix LPAR

Have you looked into the Python library for Z Open Automation Utilities (ZOAU) ? I found out about this through doing this years "Master the Mainframe". For details do a search on "zoautil-python". But I don't know whether ZOWE needs to be installed to use these (ZOWE formed a large part of MTM2020). I wrote a Python script to get details on job output (using zoautil_py.Jobs). I can send this onto you if you'd like an example of how to use the ZOAU library.

I found a post on Stack Overflow which has an example of a Python script that wraps USS commands (I didn't try it), which would be another way of accessing MVS datasets when running in OMVS.

See : stackoverflow.com/questions/59973962/utf-8-to-ebcdic-using-iconv-in-python-script-on-uss

With these two techniques you will have to specify a dataset name rather than referencing a JCL DD name.

If you need to keep the Python source code in MVS, perhaps you could have a step that copies the script to a USS directory before running it.
Back to top
View user's profile Send private message
enrico-sorichetti

Senior Moderator


Joined: 14 Mar 2007
Posts: 10727
Location: italy

PostPosted: Thu Dec 10, 2020 4:19 pm
Reply with quote

Quote:
If you need to keep the Python source code in MVS, perhaps you could have a step that copies the script to a USS directory before running it.


that' s what the script I was talking about does

I know that the forum frowns on the attachments, but since they are PDFS, You will have to live with that icon_biggrin.gif

since it looks like that there are not any IP issues here they are
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 6
Location: United States

PostPosted: Sat Dec 12, 2020 5:15 am
Reply with quote

Hello everyone,

We had our meeting with IBM today. It was a short introduction of the players, and to define what our issues are trying to replace Easytrieve in the MVS environment with Python.

They say they have been working with a few organizations on this. They only really installed Python in June 2020 in the z/OS environment from what I understood from the meeting, so this is really cutting edge for this environment.

Next week, we will be getting into more of the nuts and bolts, and I hope that I can provide everyone with more information about how this stuff works. So far, I can execute a Python program in the Unix side from JCL in the MVS side. I can open and read a Unix file and create an MVS dataset with the copied data. I can also do a LISTC command to print all the attributes to an MVS dataset. However, the SUPER frustrating issue is that I cannot figure out how to open that dataset to read it!!!!! I've spent HOURS trying various things. I have used all the methods you all have provided, which actually got me this close.

Anyways, I will continue to keep you updated. This thread will undoubtedly be searched by other developers in the future as more and more companies with z/OS systems try and shift to Python.

Thank you all,

JMitchell
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM

 


Similar Topics
Topic Forum Replies
This topic is locked: you cannot edit posts or make replies. Invoke REXX thru JCL to expand the co... CLIST & REXX 2
No new posts Invoke WebService failing with respon... CICS 3
No new posts INVOKE WEBSERVICE error CICS 11
No new posts Invoke Webservice Fails with DFHPI100... CICS 2
No new posts RESP 13 and RESP 3 : webservice invok... CICS 1
Search our Forums:

Back to Top