File Transfer does not fail when data truncated on z/OS

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

Error Message :
N/A

On z/OS if the data is truncated during a Managed File Transfer (MFT) to the z/OS machine, the transfer does NOT fail.
 
Whenever data is written into a dataset on z/OS, it uses the Record Length (LRECL) which specifies the maximum length (in bytes) of each record (line) in the dataset.

If an attempt is made to write a record into a dataset where the length is greater than the LRECL of the dataset, the system will issue an error message similar to this: 

EDC5003I Truncation of a record occurred During an I/O operation.

When a File Transfer (FT) job (object type, JOBF) is used to transfer data to z/OS, the data written into the dataset may be truncated.  If it is, the task should fail (ENDED_NOT_OK) and the File Transfer Report should contain the appropriate error message.  In Automic FT protocol version 1, it did function this way and the job failed.  However, in FT protocol version 2 such truncations are not recognized and the Task does NOT fail.  Instead the JOBF ends with an ENDED_OK status, even though records were truncated.

Automic File Transfer Protocol

The FT protocol version 1 was used with the Agent version 8 and below.  If one or both of the Agents used in the FT are lower than version 9, protocol version 1 is used automatically.
 
As of version 9 the new FT protocol (version 2) was introduced.  As soon as both Agents involved in the FT are on version 9 or higher, the newer FT protocol is used by default.
 
In addition, it’s possible to change the default by setting the protocol version with the parameter FT_Version in the Host Characteristic of the Agent: 

FT_Version = 1 means the Agent will use the old protocol.
FT_Version = 2 means the Agent will use the new protocol. Agent versions needs to be 9 or higher.

Our recommendation is to use the newer protocol, FT protocol version 2.

Investigation

Here an example to help better understand this issue. 

LRECL=9 is used for the destination dataset on z/OS, but a file with longer lines than 9 Bytes is used for the transfer.
 
JOBF with RunID 4129308 (using FT_Version 2 ) ends without an error message, but the content of the file was truncated. The Agent log doesn’t indicate a problem:

20151013/132308.977 - U2002039 Successfully established connection with '192.168.115.101:49362'  (socket handle = '4').20151013/132308.986 - U2000006 Connection to Agent 'VWGSUP11(4,ID=13)' was successfully established.20151013/132309.587 - U2000134 Thread '*FT(4129308)' started.20151013/132309.974 - U0011411 FT '4129308(oid=0)': Receiving file 'UC4SCJ.SIS.TEST.FILE1.TXT'.20151013/132320.001 - U2000135 Thread '*FT(4129308)' ended.

The Report of the File Transfer also doesn't indicate there was a problem:

2015-10-13 15:25:08 - U0011124 Selection started with filter '\\vwgsup11\c$\UC4_FT\test.file1.txt' ...2015-10-13 15:25:08 - U0011125   '\\vwgsup11\c$\UC4_FT\test.file1.txt'2015-10-13 15:25:08 - U0011126 Files selected: '1'.2015-10-13 15:25:09 - U0011133 OK    '27' Bytes, '4' Records for file '\\vwgsup11\c$\UC4_FT\test.file1.txt'->'UC4SCJ.SIS.TEST.FILE1.TXT' transferred. Duration '00:00:01'.2015-10-13 15:25:09 - U0011408 FT '4129308': FileTransfer completed.

However, lines longer than 9 characters were truncated.

<Please see attached file for image>

JOBF with RunID 4129309 (using FT_Version 1) ends with ENDED_NOT_OK,

“EDC5003I Truncation of a record occurred during an I/O operation.”.

The Agent log indicates the problem:

20151013/132506.701 - U2002039 Successfully established connection with '192.168.115.101:49450'  (socket handle = '4').20151013/132506.712 - U2000006 Connection to Agent 'VWGSUP11(4,ID=17)' was successfully established.20151013/132507.187 - U0011407 FT '4129309': FileTransfer with partner 'VWGSUP11' started.20151013/132507.303 - U0011411 FT '4129309': Receiving file 'UC4SCJ.SIS.TEST.FILE1.TXT'.20151013/132507.425 -          fwrite(...,//'UC4SCJ.SIS.TEST.FILE1.TXT') error 3: EDC5003I Truncation of a record occurred during an I/O operation.20151013/132507.425 - U0011414 Error while writing to file '//'UC4SCJ.SIS.TEST.FILE1.TXT''. Error code: '3h', error description: ''.20151013/132507.718 - U0011409 FT '4129309': FileTransfer ended abnormally.

The Report of the File Transfer also indicates the problem:

2015-10-13 15:27:07 - U0011134 ERROR '           40' bytes, incorrect transfer of '            4' records for file '\\vwgsup11\c$\UC4_FT\test.file1.txt'->'UC4SCJ.SIS.test.file1.txt'. Duration  '00:00:01'.2015-10-13 15:27:07 - U0011414 Error while writing to file '//'UC4SCJ.SIS.TEST.FILE1.TXT''. Error code: '3h', error description: ''.

The screenshot below shows the properties of the test JOBF and the Statistics of the test run.

<Please see attached file for image>

 

Cause

Cause type:
Defect
Root Cause: FileTransfer into zOS don't fail even if records have been truncated. Expected behavior is to stop transfer with ENDED_NOT_OK status and to write an error message into FileTransfer report.

Environment

OS: MVS
OS Version: N/A

Resolution

Update to a fix version listed below or a newer version if available.

Fix Status: Released

Fix Version(s):
Component(s): Agent MVS

Automation Engine 12.1.0 - Available
Automation Engine 12.0.2 - Available
Automation Engine 11.2.5 - Available
Automation Engine 11.1.6 - Available
AE 10.0.8 HF4 - Available

Additional Information

Workaround :
Use FT protocol version 1. This can be set by setting the FT_Version parameter to 1. This parameter is located in the Host Characteristics.

Note: If one or both of the Agents used for the FT is version 8 or older, the default setting for FT_Version is already 1.