Transfer to XCOM for Windows with Linux NAS file share

Products

XCOM Data Transport XCOM Data Transport - Windows

Issue/Introduction

We have a setup as described below:

XCOM for iSeries (AS400) exporting a file to a NAS File share path via a Windows 2016 server. Multiple applications access the files exported into the NAS file share.

This NAS file system is mounted as a CIFS file system on a Linux system. When viewing the file from Linux server, we see "Control-M" (^M) characters being added to each line of the file.

Could you please advise a workaround to overcome the "Control-M" characters when viewing the files from Linux as part of the XCOM transfer itself?

Environment

XCOM™ Data Transport® for Windows
XCOM™ Data Transport® for UNIX/Linux PC
XCOM™ Data Transport® for AS/400 i5/OS

Cause

The programmers of different operating systems have represented line endings using different sequences

Resolution

In a plain text file, to tell the computer that a line of text doesn't continue forever, the end of each line is marked by a sequence of one or more invisible characters, called control characters. While there are many control characters for different purposes, the relevant ones for line endings are the carriage return (CR) and line feed (LF) characters.

Unfortunately, the programmers of different operating systems have represented line endings using different sequences:

All versions of Microsoft Windows represent line endings as CR followed by LF.
UNIX and UNIX-like operating systems represent line endings as LF alone.

Therefore, a text file prepared in a Windows environment will, when viewed by a UNIX-like environment, have an unnecessary carriage return character at the end of each line.

This character will normally be invisible, though in some text editors it will show up as ^M or similar.

For TEXT type transfers, XCOM removes, unless otherwise told to via additional XCOM parameters, all EOL characters, such as CLRF and LF, from the records before sending. It is up to the receiving XCOM system to add any EOL/EOF characteristics required for that system/file system when writing to the file. For example, in Windows, both a CR and LF are required to indicate the end of a line, whereas in Linux/UNIX a LF is only required and in the HTTP protocol, the CR-LF sequence is always used to terminate a line.

For BINARY type transfers, XCOM will not remove any EOL characteristics, unless otherwise told to via additional XCOM parameters, from the records before sending nor will the receiving XCOM add any EOL characteristics when writing to the file except the last character to signify the EOF.

Therefore when receiving a TEXT type transfer to a Windows system that is writing the file to a share, it doesn't matter if it is a SMB or CIFS or Windows. Since it is Windows system receiving the file, each record will end with a CRLF. If only line feed LF is required, then the files will need to be converted from a windows text CRLF format to a Unix/Linux text LF format.

SUMMARY OF FINDINGS AND RECOMMENDED STEPS:

The original AS400 file is EBCDIC.
Running the transfer with BINARY=YES causes all lines to be concatenated into a single line i.e. no line feeds.
Running the transfer with BINARY=NO enables line feeds to give required multiple lines but ^M characters are visible at the end of each line when viewed from a Linux server.
The above XCOM behaviours are working as designed per the explanation above.
With the option BINARY=NO.
Due to differences between Windows and Linux EOL characters (CRLF for Windows versus just LF for Linux), to avoid the ^M carriage return (CR) characters it is recommended to use XCOM for Linux instead of XCOM Windows.
If #2 is not possible, the only option to remove ^M characters under XCOM for Windows would be to use an XCOM exit (using Perl or PowerShell or some other utility program e.g. Windows version of dos2unix) to convert the line delimiters to just the required LF. XCOM exits (Processing scripts) are documented here and either XPPCMD or XENDCMD could potentially be used:
Use the Processing Scripts

Also see this KB article: Using the different CA XCOM pre and post processing scripts
NOTE: Some of the applications/programs on Windows read the files differently when they use only LF as a delimiter. If NAS storage is being accessed by Windows/Linux/Unix environments, there could be side effects from changing the record delimiters.