An Initiated secured AT-TLS transfer from z/OS to Windows fails with message “XCOMN0780E Tcpip 503: Received simulated FMH-5 record w/ bConvType = 208, not 209.” The messages received on the z/OS side are:
EZD1287I TTLS Error RC: 9 Initial Handshake
XCOMM1467E OMVS Error: EDC5140I Broken pipe. (errno2=0x77F37242)
XCOMM0780E Txpi 215: Socket send error return value = 140
Note: This also happens when transferring from XCOM for z/OS to XCOM for Linux
1. Apply the following fixes:
LU02250 – XCOM r11.6 for Windows
LU02422 – XCOM r12 for z/OS
LU02343 – XCOM r11.6 for Linux
2. Change the DH_2048_RECEIVE_SIDE = parameter as shown below in the configssl.cnf for Windows and/or Linux:
# Optional (for RSA NO, see above). If RSA NO and DH files empty,
# then internal program tables will be used.
[DH]
DH_512_RECEIVE_SIDE =
DH_1024_RECEIVE_SIDE =
DH_2048_RECEIVE_SIDE = %XCOM_HOME%\\Config\\dh2048.pem
DH_4096_RECEIVE_SIDE =
When RSAKEY for the RECEIVE_SIDE is set to "NO" and there are no DH key files provided, XCOM will use an internal key which may cause a problem. The internal key can be quite small by modern standards and since this is negotiated, AT-TLS should have no problems using the longer key. This is also, by definition, more secure.
These particular parameter will ONLY affect the RECEIVE_SIDE (i.e. when the Windows XCOM is acting as a server). That is also likely why it works when Windows is functioning as the client.