On a TEST environment transfer of dataset with CODE=EBCDIC file from XCOM for z/OS 12.0 to XCOM for Linux 12.0 (r12.0 22042 SP00 64bit) with COMPRESS=NO is working.
However when use COMPRESS=YES (defaults to mode ZLIB) the transfer of the same dataset fails with this message:#XCOMU0503E Invalid compressed data
On the PROD environment both transfers are successful.
XCOM™ Data Transport® for z/OS
XCOM™ Data Transport® for Linux
On the TEST environment the SYSIN01 parameters show record packing (PACK=LENGTH) is being used and logs showed MAXPACK=1023000 is used from the CONFIG member.
After testing by adding MAXPACK=31744 to the SYSIN01 parameters to override the CONFIG MAXPACK value, the transfer with COMPRESS=YES (mode ZLIB) was successful.
On the PROD environment which did not have the problem MAXPACK=31744 was already set in the CONFIG member.
After capturing a trace on the z/OS side when using MAXPACK=1023000 it showed that the very first buffer being sent to the Linux server contains a single record with its length in the first 4 bytes (x'00000094') which is less than 500 bytes. This record is not being compressed on the z/OS side because it is too short. The XCOM for Linux server is not honoring the code to check if the length is > 500 bytes and is incorrectly trying to decompress it which causes the error.
Testing in-house with the same scenario and latest maintenance applied to XCOM for Linux 12.0 shows the problem does not occur.
The actual fix for this problem was made in this earlier patch LU05970 ("Transfer of files under 500 bytes fail on partner when using compression").
To be able to use any MAXPACK value on z/OS the resolution is to apply the latest XCOM for Linux patch which can be accessed from the Support Portal XCOM for Linux 12.0 Solutions page
(XCOM patches for distributed OS platforms contain cumulative fixes)
XCOM doc. reference: The Batch Interface > SYSIN01 Parameters > General Parameters, H-P