The RA-FTP Solution in combination with a Mainframe based FTP Server

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

The RA-FTP Solution in combination with a Mainframe based FTP Server

The graphical user interface of the RA-FTP Solution (further referred to as the GUI) does not fully support the specifics of certain Mainframe (MVS, z/OS, OS390, AS400 and Siemens BS2000) file systems. This does not hinder users deploying the RA-FTP Solution for transferring files from or to those mainframe systems. But it requires some knowledge of the file system and its specifics. By specifying correct file names in the various fields used for job definition, a user can successfully create jobs to do the task at hand. This article will give an explanation of the specifics of this file system and how certain ftp actions act specifically within this file system.
 

Basic mainframe file system knowledge

This article is not intended to give a full introduction to the mainframe file system. But it will point out some theories that might be necessary for a user of the RA-FTP solution to be aware of. There are various sources of information available on the internet that can be used to get deeper knowledge whenever required. In many cases the user might already have this knowledge or has access to mainframe operators to get this information.
Actually there are two file systems available on mainframe systems

1. The POSIX-style zFS former called HFS.
   This is a “relatively new” file system that is still not readily available. It is POSIX-like which means similar to UNIX and therefore no problem at all. It can be accessed through a mainframe FTP-Server and as soon as the user enters its root path in this file system they should have no difficulties with our RA-FTP Solution GUI.
   
   There still can be an initial problem directly after the connection is established. Normally there is only one FTP-Server running on the mainframe and if it is configured to put the user in the regular MVS file system after login a user might consider some problems within the GUI. They will need to enter the zFS (HFS) path first manually to access the zFS.
   
   This article will not go deeper into zFS but will focus on an older and much more frequently used MVS file system. Note that all further information such as having no directories does not refer to zFS.
    
2. The regular MVS, BS2000 and AS400 file systems.
   This is most frequently used on mainframe systems. There is, in many cases, a large amount of very old legacy programs running that need to be fed with data and those applications do not have access to file systems other than the regular MVS file system. So a common application for our RA-FTP solution is to store data on a mainframe system for further processing by this legacy applications or to have processed data returned back to other systems by using RA-FTP to get the data onto a Windows or UNIX system.

 

The MVS file system

There are no directories!
Most users know about Windows or UNIX file systems. One important feature of these file systems are directories. This concept does not exists in the base file system of a mainframe system.
Dataset
A dataset can be seen as a file. Datasets have various parameters associated with them. Basically these parameters define the format and size. Applications on mainframes strongly relate to the format of their input data. Prominent examples of this formats are:

• Fixed block - each line has a fixed length
• Variable block - each line has variable length
• Partitioned - contains “members”
• Library – used for executables

So, in many cases, it is important to create a file in the proper format. The mainframe ftp server creates per default files in variable block format and by using the “Make Directory” command a partitioned dataset. The format of the created files can be changed by specific “Quote Site” commands. It is important to know that this is possible. For details please refer to the documentation of the mainframe ftp. It can be found on the internet or in the ftp server manual.
 

Dataset names

A valid dataset name consists of qualifiers separated by dots (.) and must not exceed 44 bytes in total length. Each qualifier must start with alphabetical character (not a number).
TEST.RAFTP.INPUT.X1 is a valid data set name. It consist of the qualifiers TEST, RAFTP, INPUT and X1. Qualifiers are important on a mainframe system as they can be used to structure data. So datasets can be listed by using * as wildcard character. So listing datasets with the pattern TEST.RAFTP.* will list all datasets starting with the two qualifier TEST and RAFTP. Note that this can simulate some sort of directory structure but still there are no directories on mainframes at all.
A special meaning is associated with the term “high level qualifier”. This means the first qualifier in a dataset name. It can be compared with the entries in the root folder of a UNIX-like file system where each first level folder has some special meaning to the operating system.
 

Full qualified data set name

Using a dataset name such as TEST.RAFTP.INPUT.X1 is treated by the mainframe system as not fully qualified and will get prefixed internally by a certain high level qualifier as it is defined by the system operator. Usually this is the user ID of the login user. So the above example, assuming the user ID is USER1, will be prefixed by the system and become ‘USER1.TEST.RAFTP.INPUT.X1’. This is called a full qualified data set name.
When using RA-FTP to connect to a mainframe ftp server we have to know that all dataset names are treated like the above example but there is a way to get around the problem by enclosing the dataset name in single quotation marks.
So for our above example TEST.RAFTP.INPUT.X1 equals ‘USER1.TEST.RAFTP.INPUT.X1’ assuming the login users ID is USER1. In many cases a user will need to store or retrieve data having a different high level qualifier then its user ID. If it has sufficient rights to access this data it can be done by using the fully qualified data set name in RA-FTP Job entry fields, so enclosed in single quotation marks.
Note that the access right system (RACF) on the mainframe system regulates access rights to files based on this qualifier system. So the system operator might give access to a certain user or group to ‘SYSTEM.INCLUDE.*’. So the user used to connect to the ftp server must have access rights to the files they want to read or write.
 

Partitioned Datasets

This type of dataset can contain members. The rules for dataset names also apply to this type. A partitioned dataset can be compared to a certain degree to a directory. The members could be seen like files in a directory. But there are limitations for example a partitioned dataset can’t contain other partitioned datasets. The member names can’t be longer than 8 characters and it must start with an alphabetic character (no numbers at first position).
Members of portioned datasets can be access directly by specifying their name enclosed in brackets right after the dataset name. e.g.: TEST.RAFTP.INPUT.PS(MEMB1). Listing of members can be done by using * as wildcard in the member name. e.g.: TEST.RAFTP.INPUT.PS(*) will list all members or TEST.RAFTP.INPUT.PS(ME*) will list all members having a name starting with ME.
This rounds off a very short introduction to the mainframes file system. More details can be found on the internet and in the IBM manuals e.g. ISPF manual. But these basics should be enough to get user’s jobs out of the trouble whenever the target ftp server is a mainframe ftp server. The next chapter will point out the specific differences in common ftp commands.
 

Specific behavior of a mainframe FTP Server

This unit will provide information on the most important and annoying specifics of the mainframe ftp server based on the very specific MVS file system. It is not a complete list but will be enough to help to get out of problems with standard daily ftp business such as creating directories, up & downloading files.
The Working Directory Dataset Name Prefix is used on mainframe ftp servers instead of a working directory because there are no directories.

• cd (change directory) changes the Working Directory Datasetname Prefix but also opens a partitioned dataset
• ls (list directory) will list datasets using the Working Directory Datasetname Prefix
• pwd (print working directory) will print the Working Directory Datasetname Prefi

E.g.:
Let’s assume we have those datasets available in our root directory and our user ID is USER1. There are written below in the Full Qualified Dataset Name notation.

‘USER1.TEST.CITY.VIE.MAP1’
‘USER1.TEST.CITY.VIE.MAP2’
‘USER1.TEST.CITY.NY.MAP1’
‘USER1.TEST.CITY.NY.MAP2’

After we have logged on to the ftp server it will change our working directory name prefix to ‘USER1.’.
If we issue the command pwd the ftp-server will reply with "USER1." is working directory.
Issuing the command ls will result in a list of:

TEST.CITY.VIE.MAP1
TEST.CITY.VIE.MAP2
TEST.CITY.NY.MAP1
TEST.CITY.NY.MAP2

Although there are no directories we can use the Change Directory command to extend our working directory name prefix. Issuing the command cd test.city.vie will do that. Issuing pwd results in a message: ‘USER1.TEST.CITY.VIE.’ is working directory and ls will be responded to by the list:

MAP1
MAP2

We can use this to limit our list of datasets but we have to keep in mind that there are no directories even when the command Change Directory implies this. In fact the previous list is just a representation of the 2 datasets:

‘USER1.TEST.CITY.VIE.MAP1’
‘USER1.TEST.CITY.VIE.MAP2’

Still there is some use or specific to this as it will allow us to filter. But, it also has the effect of changing the default names for a Get operation. So issuing a command get MAP1 will result in a file named MAP1 on the local system with the content of ‘USER1.TEST.CITY.VIE.MAP1’.
You can use cd .. to navigate the working directory name prefix backwards. It will change our current working directory prefix to ‘USER1.TEST.CITY.’ and issuing the ls command will lead to the following response:

NY.MAP1
NY.MAP2
VIE.MAP1
VIE.MAP2

Consequently get VIE.MAP1 will result in a file named VIE.MAP1 on the local system having the same content as MAP1.
There is of course one exception which should be mentioned. When using cd on a portioned dataset (those with the members inside) it will kind of “open” that dataset and a subsequent ls will display the members of this dataset. Get operations will then copy the members to the local system and Put operations will add a new member to the dataset.
New partitioned datasets can be created by using the command mkdir. So mkdir MAPS will create a partitioned dataset and as our current working directory is still ‘USER1.TEST.CITY’ the fully qualified dataset name is ‘USER.TEST.CITY.MAPS’. An ls command will respond

NY.MAP1
NY.MAP2
VIE.MAP1
VIE.MAP2
MAPS

We can change into maps by issuing the command cd maps and copy a local file to it by using the Put command. It will create a member within the partitioned dataset.
This concludes the specifics of the mainframe ftp as those mentioned lead mostly to problems and misunderstandings. There is of course a lot more such as various Quote Site commands to steer the type and sizes of the created datasets, the code page conversion and much more. You can find a lot of information on this topic by searching the internet. IBM websites often have manuals online with complete sets of information about their products and the mainframe ftp server is an IBM product.
 

Environment

Release: AOMVMW99000-4.0-Automic-Operations Manager-VMware vCenter Agent
Component:

Resolution

RA-FTP Solution GUI and Mainframe FTP-Server specifics

Using the GUI

When working with the RA-FTP Solution GUI in relation to a mainframe FTP-Server the GUI will not respond properly to actions in the file browser window because of the different file system architecture. The GUI will display portioned datasets as directories but double clicking it will not have the effect of “entering” the directory. The screenshot to the right shows four directories that are in fact portioned datasets. The path field (marked in yellow) shows a very strange entry as the result of a double click. Additionally the RA-FTP Agent reports an error in its logs as this file name is clearly rejected by the MVS FTP Server as it is not a valid dataset name. In addition the Last Mod Time and the Size fields contain invalid data

<Please see attached file for image>

Currently the RA-FTP GUI does not support the MVS file system but it will allow you to enter your path in the path field manually. You can use this fact to enter a valid dataset name into the path field. So if you enter the dataset name that is in fact the name of the partitioned dataset it will apply a filter but still it does not enter the partitioned data set and you do not see the members.

<Please see attached file for image>

 

You can work around this by entering the search pattern manually and that would be for showing all members TEST.CITY.MAPS(*). This will show you the list of all members.
The file browser still doesn’t give you any information on the file size and change dates but it will allow you to figure out a particular member name or dataset name to use it in your job definition. The file browser itself is not necessary for the job execution at all. It is a helper in defining jobs, getting proper path information from local file system and remote ftp server but it is not used when the job runs automated.

<Please see attached file for image>

 

Defining Jobs

Defining Jobs to use for data transfers from or to mainframe ftp servers should be done by using the free form command only for all commands on the mainframe ftp server. You can use any other command for the local system or another ftp connection as long as this is not a mainframe ftp.
The command sequence to the right will create a local directory, change locally into it and transfers two datasets from the mainframe to the local system. It will then create a portioned dataset on the mainframe and copy the two files back to the server but now as members into the newly created portioned dataset.
You can use any ftp command on the mainframe but you need to know about the specific behavior of the file system as well as the specific behavior of the commands as described in the previous units.
In this example I have used both fully and partially qualified dataset names. I have used them to create a portioned dataset and I have transferred files from the mainframe to a local Windows folder and back to the mainframe. This job will run without any problems.

<Please see attached file for image>

 

Conclusion

So it is possible to use the RA-FTP Solution in collaboration with a mainframe ftp server. It has its limitations as it is not supported fully yet. It is necessary for the user to learn the very basics of the MVS file system and the specific behavior of certain ftp commands which are sometimes confusing and very different from its “standard” behavior on Windows or Unix Systems. But with a little knowledge about these specifics a user might create jobs within the solution to transfer the data they need.