IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

JCL to invoke Python in z/OS


IBM Mainframe Forums -> JCL & VSAM
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 7
Location: United States

PostPosted: Tue Dec 08, 2020 4:05 am
Reply with quote

Good afternoon. I am heading up a project in our department to convert several hundred Easytrieve Plus programs to Python. The advantages are numerous. Python is an interpretive language like Easytrieve, there are no licensing fees, there are a bunch of kids coming out of school with that skill set....on and on.

We have Python 3.8.5 installed in the USS partition of our z/OS IBM mainframe.

My JCL to invoke Python using IBM's BPXBATCH module is:

Code:
//xxxxxMH1 JOB (11111,1),'PYTHON ',TIME=(,15),
//         NOTIFY=&SYSUID,MSGLEVEL=(1,1),       
//         REGION=0M,                           
//         CLASS=T                               
/*ROUTE    PRINT Q                               
//***********************************************
//STEP10   EXEC PGM=BPXBATCH                     
//STDPARM  DD DSN=XXXXXMH.TEST.PYTHON(PYPATH),   
//            DISP=SHR                           
//         DD DSN=XXXXXMH.TEST.PYTHON(PYVER),   
//            DISP=SHR                           
//STDOUT   DD SYSOUT=*                           
//STDERR   DD SYSOUT=*


In the PYPATH file, the single line invokes Python using the following string:
sh /usr/lpp/IBM/cyp/v3r3/pyz/bin/python3

In the PYVER file, I put in this line:
Code:
--version


This runs perfectly in batch. The STDOUT DD produces this 1 line report:
Python 3.8.5

However, if I change the PYVER file to include a hello world statement like this:
Code:
--version
print('Hello world')


I only get the STDERR DD producing this:
Code:
FSUM7332 syntax error: got (, expecting Newline


I believe the problem is the ASCII/EBCDIC translation when going to the USS partition.

I've tried just about everything I can think of. I've tried setting codepages as recommended by Python, but those require a # to begin the statement, and when I do that, Python can't see any statements after that. It may be theat the hashtag is not translated correctly. However, since the first example, with only the Python --version statement comes back clean, I believe I am hooking into the Python interpreter correctly.

I've tried to get the Systems Programmers to fix it, but they know nothing about it. We have a 30 minute call with IBM in a week, but this is crazy. I can't be the first guy to be doing this.

I find NO information on the Internet about Python's JCL in z/OS, but people are apparently running it in the USS partition.

Any insight would be enormously appreciated.

JMitchell

Please use the code tags when posting code and data - done for you this time
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1334
Location: Bamberg, Germany

PostPosted: Tue Dec 08, 2020 4:28 am
Reply with quote

While STDPARM is able to accept a --version parameter, I don't think this applies for Python code itself. You could try that via STDIN.
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 7
Location: United States

PostPosted: Tue Dec 08, 2020 5:32 am
Reply with quote

Joerg,

Thanks for that info. I added the STDIN DD statement in the JCL and created the one line file with the print('Hello world') statement. It died with the the same error. This is good news somewhat. Now I know that I need to put the program in the STDIN statement.

I also tried a line ahead of the print statement that was # coding=IBM-1047. As before, once Python saw the hashtag, it ignored the rest of the code, so the print statement was bypassed.

I do appreciate your input very much.

JMitchell
Back to top
View user's profile Send private message
Joerg.Findeisen

Senior Member


Joined: 15 Aug 2015
Posts: 1334
Location: Bamberg, Germany

PostPosted: Tue Dec 08, 2020 1:01 pm
Reply with quote

You might have a look at ibm.com how to set up STDENV for a Locale. May it be of help.
Back to top
View user's profile Send private message
WilliamDownie

New User


Joined: 01 Jul 2020
Posts: 21
Location: UK

PostPosted: Tue Dec 08, 2020 2:12 pm
Reply with quote

I've been tinkering with running OMVS Python scripts from from z/OS. I've just started looking into this so may not be the correct way of setting things up, but it may help you .

The error you have got looks like print('Hello world') is not being interpreted as Python.

What I've done to run a OMVS Python script from z/OS is as follows:

JCL
Code:
                                                         
//STEP1    EXEC PGM=BPXBATCH                                     
//STDPARM  DD   DISP=SHR,DSN=Z01408.SCRIPTS(RUNSH)               
//STDOUT   DD   SYSOUT=*                                         
//STDERR   DD   SYSOUT=*                                                                                                                                                                                                       

Z01408.SCRIPTS(RUNSH)
Code:

sh /z/z01408/hello.sh

hello.sh
Code:

#!/bin/sh
python3 hello.py


hello.py This resides in /z/z01408
Code:

print("hello world")

Output from STDOUT is:
Code:

******************* TOP OF DATA ******
hello world 
****************** BOTTOM OF DATA ******


Note, if I change "Z01408.TEST.PYTHON(RUNSH)" to "sh /z/z01408/hello.py" I get this error:

Code:

/z/z01408/hello.py 1: FSUM7332 syntax error: got (, expecting Newline 


which is why I think , in your example, your script is not being interpreted as Python.
Back to top
View user's profile Send private message
Apoorva

New User


Joined: 28 Jan 2020
Posts: 49
Location: India

PostPosted: Tue Dec 08, 2020 4:03 pm
Reply with quote

WilliamDownie wrote:

which is why I think , in your example, your script is not being interpreted as Python.


Informative. Thanks for the info!
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10886
Location: italy

PostPosted: Tue Dec 08, 2020 4:41 pm
Reply with quote

Quote:
sh /z/z01408/hello.py


You just told sh to interpret a python script

sh in this case disregards the shebang line

it is like expecting the cobol compiler to understand a fortran source
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 7
Location: United States

PostPosted: Wed Dec 09, 2020 6:17 am
Reply with quote

@Joerg.Findeisen, @WilliamDownie, @Apoorva, @enrico-sorichetti

You are all Rock Stars!!!!!!

Using information from you all, I was able to get my first Python program running from a batch environment on the mainframe. Below is the JCL and members that I used:

Code:
//xxxxxMHP JOB (#####,1),’PYTHON’,TIME=(,15),
//         NOTIFY=&SYSUID,MSGLEVEL=(1,1),       
//         REGION=0M,                           
//         CLASS=T                               
/*ROUTE    PRINT Q                               
//*****************
//STEP10   EXEC PGM=BPXBATCH                     
//STDPARM  DD DSN=xxxxxMH.TEST.SCRIPTS(RUNSH), 
//            DISP=SHR                           
//STDOUT   DD SYSOUT=*                           
//STDERR   DD SYSOUT=*


In my xxxxxMH.TEST.SCRIPTS(RUNSH) file, there is a single line where IBM recommends the installation of Python, which our System Programmers followed, followed by a space, then the Python source code name:
sh /usr/lpp/IBM/cyp/v3r8/pyz/bin/python3 /tmp/hello.py


In the ISPF 3.17 menu, I access the z/OS UNIX Directory List. I created the hello.py member in the /SYSTEM/tmp directory. This is the file:

Code:
print('Hello world')   
a = 4                   
b = 6                   
c = a+b                 
print('a = '+str(a))   
print('b = '+str(b))   
print('a + b = '+str(c))     


I was delighted to see this simple output:
Hello world
Code:
a = 4         
b = 6         
a + b = 10   

It looks pretty simple when you see it, but I ran hundreds of iterations of trials and errors to get to this point.

My next challenge is to be able to assign the source code to a dataset that is in the STDIN DD statement for the invoking batch program BPXBATCH. If anyone has suggestions, I am very grateful.

I really do appreciate you all taking your time to help me all. Collaboration across the globe!

JMitchell
Back to top
View user's profile Send private message
Apoorva

New User


Joined: 28 Jan 2020
Posts: 49
Location: India

PostPosted: Wed Dec 09, 2020 11:28 am
Reply with quote

jmitchell wrote:
@Joerg.Findeisen, @WilliamDownie, @Apoorva, @enrico-sorichetti

You are all Rock Stars!!!!!!
JMitchell


Glad to know that it worked! I am no "Rock Star" because I didn't provide any inputs icon_smile.gif. I was carefully following this post because I have also been running a Python application (on distributed server) that integrates JIRA with the Mainframes (basically captures error messages in z/os, and logs it in JIRA as issues). If I could run my python application on z/os (like you did) I could probably deploy that python application on z/os itself.
Back to top
View user's profile Send private message
WilliamDownie

New User


Joined: 01 Jul 2020
Posts: 21
Location: UK

PostPosted: Wed Dec 09, 2020 6:47 pm
Reply with quote

Quote:
My next challenge is to be able to assign the source code to a dataset that is in the STDIN DD statement for the invoking batch program BPXBATCH

The following doesn't assign the source code to STDIN, so maybe not what you are after, but as I spent some time looking into it, I thought I'd post incase it gives you some ideas.

According to the documentation I've found BPXBATCH can run a shell script or an executable program. In your case you are running a shell script to invoke the Python script. The manual says "... use STDIN to define the name of the shell script to be invoked."

You could point to the script in STDIN:

Code:

//STEP1    EXEC PGM=BPXBATCH                                         
//STDIN    DD   PATH='/z/z01408/directory1/hello',PATHOPTS=(ORDONLY) 
//STDOUT   DD   SYSOUT=*                                             
//STDERR   DD   SYSOUT=*                                             


where , in my case, hello is:
Code:
python3 /z/z01408/hello.py

and hello.py is
Code:
print("hello world")

Running this output the "hello world" to STDOUT.

Note, in the above I did not need to specify this was a shell script (using "sh") because that is the default.

It look me a while to find the IBM manual (I couldn't find it on the IBM web site). So, if you'd like me to forward it to you let me know. I find the manual easier to use that IBM's online resource (as I find clicking from page to page annoying).
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 7
Location: United States

PostPosted: Thu Dec 10, 2020 6:51 am
Reply with quote

First, I want to apologize to @enrico-sorichetti for not including you in the Rock Star category. You really are. I was under the impression the --version statement was interpreted by Python. Then I ran it in Juptyter and it didn't recognize it as a Python command. That led me to not using this as a verification of the connection to Python.

So, my dilemma is that my organization needs to run batch programs from an IBM z/OS MVS environment that used to use Easytrieve, and wants to switch to Python. We run these jobs using JCL. There is no Python program that IBM has installed in the MVS environment. They install it in the z/OS Unix environment, which is a separate LPAR located next to the MVS LPAR.

Using IBM utility program BPXBATCH, I can set a parameter to activate Python in the next door Unix environment, and pass in the name of the Python program I want it to execute. That program must exist (as far as I know) inside the Unix world. The reports are returned to the MVS world thru the DD STDOUT.

The problem is, I need the program to be able to live in the MVS environment as a dataset member, so it conforms to our traditional change control management requirements. It also must be able to read unlimited various files and formats from many DD statements, as well as write various unlimited files. In our environment, our programs may read/write 100-500 million records here or there. This is no Mom and Pop shop.

I have been able to create a Python stub program in the Unix LPAR that opens and executes another file containing the desired Python program. However, that called Python program is also existing in the Unix LPAR. I need to be able to call a specific DD statement in the MVS JCL that will contain the path to the program. For Easytrieve, it was the SYSIN DD.

I cannot even figure out how to open any MVS DD files from the Unix LPAR. I have found that in Python 2.7 there was a ddreader.py, but I can't find any further references than that.

I'm pretty exhausted with how difficult this is. I would think IBM would have supplied a regular program, so in your JCL you could specify PGM=PYTHON and have access to all the IBM file services.

Any encouragement, hope, or insight is appreciated.

JMitchell
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10886
Location: italy

PostPosted: Thu Dec 10, 2020 2:34 pm
Reply with quote

You have nothing to apologise about icon_biggrin.gif

googling around I found this link

www.linkedin.com/pulse/running-unix-python-perl-php-scripts-via-jcl-zos-jimmy-spijker?trk=public_profile_article_view

the script and the jcl are unloadable as PDFs

even if the script is pretty complicated , it is worth exploring

good luck
Back to top
View user's profile Send private message
WilliamDownie

New User


Joined: 01 Jul 2020
Posts: 21
Location: UK

PostPosted: Thu Dec 10, 2020 4:00 pm
Reply with quote

Quote:
I cannot even figure out how to open any MVS DD files from the Unix LPAR

Have you looked into the Python library for Z Open Automation Utilities (ZOAU) ? I found out about this through doing this years "Master the Mainframe". For details do a search on "zoautil-python". But I don't know whether ZOWE needs to be installed to use these (ZOWE formed a large part of MTM2020). I wrote a Python script to get details on job output (using zoautil_py.Jobs). I can send this onto you if you'd like an example of how to use the ZOAU library.

I found a post on Stack Overflow which has an example of a Python script that wraps USS commands (I didn't try it), which would be another way of accessing MVS datasets when running in OMVS.

See : stackoverflow.com/questions/59973962/utf-8-to-ebcdic-using-iconv-in-python-script-on-uss

With these two techniques you will have to specify a dataset name rather than referencing a JCL DD name.

If you need to keep the Python source code in MVS, perhaps you could have a step that copies the script to a USS directory before running it.
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10886
Location: italy

PostPosted: Thu Dec 10, 2020 4:19 pm
Reply with quote

Quote:
If you need to keep the Python source code in MVS, perhaps you could have a step that copies the script to a USS directory before running it.


that' s what the script I was talking about does

I know that the forum frowns on the attachments, but since they are PDFS, You will have to live with that icon_biggrin.gif

since it looks like that there are not any IP issues here they are
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 7
Location: United States

PostPosted: Sat Dec 12, 2020 5:15 am
Reply with quote

Hello everyone,

We had our meeting with IBM today. It was a short introduction of the players, and to define what our issues are trying to replace Easytrieve in the MVS environment with Python.

They say they have been working with a few organizations on this. They only really installed Python in June 2020 in the z/OS environment from what I understood from the meeting, so this is really cutting edge for this environment.

Next week, we will be getting into more of the nuts and bolts, and I hope that I can provide everyone with more information about how this stuff works. So far, I can execute a Python program in the Unix side from JCL in the MVS side. I can open and read a Unix file and create an MVS dataset with the copied data. I can also do a LISTC command to print all the attributes to an MVS dataset. However, the SUPER frustrating issue is that I cannot figure out how to open that dataset to read it!!!!! I've spent HOURS trying various things. I have used all the methods you all have provided, which actually got me this close.

Anyways, I will continue to keep you updated. This thread will undoubtedly be searched by other developers in the future as more and more companies with z/OS systems try and shift to Python.

Thank you all,

JMitchell
Back to top
View user's profile Send private message
Hungryhead

New User


Joined: 21 Apr 2022
Posts: 1
Location: India

PostPosted: Thu Apr 21, 2022 3:45 pm
Reply with quote

Hi Mitchell,

I am trying similar thing for our application.

Have you got further learning or updates from your IBM team?

I would love to hear it and explore it.
Back to top
View user's profile Send private message
jmitchell

New User


Joined: 02 Oct 2020
Posts: 7
Location: United States

PostPosted: Thu Apr 21, 2022 8:27 pm
Reply with quote

Hungryhead,

I'm sad to report that I wasn't able to crack that nut. Other project priorities took away the time to continue this research. If you finally triumph over this challenge, please share.

Thanks, and good luck!

Mitch
Back to top
View user's profile Send private message
RonnyFunk

New User


Joined: 25 Apr 2022
Posts: 2
Location: Germany

PostPosted: Mon Apr 25, 2022 11:39 am
Reply with quote

Hi,

just in case your dataset read problem has not been solved already. You actually need to use the python package provided with the IBM Z Open Automation Utilities. Below is a quick and dirty example of how to do that.

In case you need to install the utilities you can find the doc here and the Python api reference is here

greetings

Ronny

Code:

#######################################################################
# Example IBM Enterprise Python and IBM Z Open Automation Utlities    #
#######################################################################
import sys,os
import time
import argparse
from zoautil_py import datasets, mvscmd
from zoautil_py.types import DDStatement, DatasetDefinition, FileDefinition
################################################################################
# Parse passed arguments                                                       #
################################################################################
def parse_our_args(argv=None):
    program_name = os.path.basename(sys.argv[0])
    if argv is None:
        argv = sys.argv[1:]

    try:
        parser = argparse.ArgumentParser(program_name)
        parser.add_argument("-dsn", "--dataset-name", help="Dataset Name", required=True)

        opts = parser.parse_args(argv)

        return opts

    except Exception as e:
        indent = len(program_name) * " "
        sys.stderr.write(f'{program_name}:{repr(e)}\n')
        sys.stderr.write(f'{indent} for help use --help\n')
        sys.exit(16)

def main(dsn):
    if datasets.exists(dsn):
        print(datasets.read(dsn))
        rc = 0
    else:
        print(f'Dataset {dsn} not found')
        rc = 12
    return rc

if __name__ == '__main__':
    args = parse_our_args()
    retcode = main(args.dataset_name)
    sys.exit(retcode)

Back to top
View user's profile Send private message
hankoerlemans

New User


Joined: 25 Jan 2018
Posts: 61
Location: Australia

PostPosted: Tue Apr 26, 2022 7:20 am
Reply with quote

Umm never saw this post so I went with (some time ago) IEBGENER to output an INSTREAM python program into a temp sequential file.

Second IEBGENER to output a Rexx program to a temp PDS for a TSO step.
I find most OMVS stuff easiest to do with Rexx BTW.

So one of things the Rexx does is

Code:

/* This is easier than trying to work out how to get     
   newlines added via IEBGENER or some such */           
Address TSO "execio * diskr msglist (stem lines. finis"   
'writefile msglist.py 700 lines. 0'                       


After the file exists you can "chtag -tc 1047 msglist.py;" to make sure it's interpreted properly. I happened to do that with a spawn of /bin/sh .

Elsewhere I invoke said msglist.py with

Code:

/* Python to parse the list */                                 
"pipe so."                                                     
"pipe se."                                                     
map.1 = so.2                                                   
map.2 = se.2                                                   
arg.3 = 'python3 msglist.py'                                   
"spawnp (arg.1) 3 map. arg. env."                             
pid = retval                                                   
if pid < 1 then do; Say 'Spawn failed for python3';exit 99;end;
"wait wst."                                                   
"close "so.2                                                   
"close "se.2                                                   
say 'STDOUT:'                                                 
fd = so.1                                                     
call pythonout                                                 
say 'STDERR: 'buf                                             
fd = se.1                                                     
call pythonout                                                 
"close "so.1                                                   
"close "se.1                                                   
if wst.w_exitstatus > 0 then do                               
  say 'STDERR: python3 ended with RC='wst.w_exitstatus         
  exit 99                                                     
end                                                           


So execution really starts with a batch TSO step invoking the Rexx which drives OMVS shell and python.

Possibly a bit convoluted but it helped with my investigations and prototyping the python code quickly. i.e. ISPF edit is quick for me :-)
Once stable it can be stored somewhere useful.
Back to top
View user's profile Send private message
RonnyFunk

New User


Joined: 25 Apr 2022
Posts: 2
Location: Germany

PostPosted: Tue Apr 26, 2022 9:42 am
Reply with quote

HungryHead,

This is not only a bit convoluted it basically is in the back, through the chest in the eye icon_lol.gif The problem is, IBM does not promote IBM Z Open Automation Utilities etc. enough. Every z Shop should know about them, where to find the documentation and how to use them with Shell Scripts and with IBM Enterprise SDK for Python.As an example here is a PDS Compress in Place from inside z/Unix (USS) Shell (bash in this case). SYSIN comes from stdin, SYSPRINT goes to stdout:
Code:
echo " COPY OUTDD=DD2,INDDD=DD1" | mvscmdauth --pgm=iebcopy --DD1=MY.PDS,SHR --DD2=MY.PDS,SHR --SYSIN=stdin –SYSPRINT=*
The RC from IEBCOPY is available inside the Shell $?

The utilities are one piece of the software required to make z/OS way more modern. They are hiding everything behind wired terms like IBM Wazi Developer for Code Ready Work Spaces. But every z Shop can modernize their z/Unix environment today and could have started at least two years ago. It does not require Openshift or the like. Basically everything you need is already part of z/OS or can be downloaded directly using your IBM Id or from IBM Shop Z as pax or SMP/E installs.

Here is what you need to do:

z/OS

1) Install ported software from Rocket using miniconda (bash, git , make etc)
2) Install IBM Enterprise SDK for Python and IBM Enterprise SDK for Go and IBM Enterprise SDK for Nodejs
3) Install IBM Z Open Automation Utilities
4) Customize your /etc/profile (very important)
5) Customize z/OSMF REST API
6) Customize z/OS NFS Server
7) If your shop does fullfill the requirements for z/OS Container Extentions (z/CX) use z/OSMF Workflows to customize it

Desktop Software

1) Install Nodejs
2) Install MS VS Code and use it to install the following VS Code Extensions:
    IBM Z Open Editor
    Zowe Explorer for MVS


3) Install ZOWE CLI and at least the Secure Credential Store for ZOWE CLI
4) Install MS Windows Terminal

With this in place you will have a modern development experience attractive to young developers and old but young minded sysprogs (as myself) icon_lol.gif.

greetings

Ronny
Back to top
View user's profile Send private message
hankoerlemans

New User


Joined: 25 Jan 2018
Posts: 61
Location: Australia

PostPosted: Tue Apr 26, 2022 11:01 am
Reply with quote

No doubt you are correct.
I will defer to you on this but I never had much time for the z/OSMF monolith.
I'm happy with my nuts and bolts for what I need.
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> JCL & VSAM

 


Similar Topics
Topic Forum Replies
No new posts Error to invoke MPP program through B... IMS DB/DC 3
No new posts Reading dataset in Python - New Line ... All Other Mainframe Topics 22
No new posts Invoke stored procedure via batch JCL. DB2 2
No new posts Invoke IMS transaction from .NET IMS DB/DC 1
This topic is locked: you cannot edit posts or make replies. Invoke REXX thru JCL to expand the co... CLIST & REXX 2
Search our Forums:

Back to Top