XCOM SSL handshake fails with XCOMU0298E and XCOMU0780E

Products

XCOM Data Transport XCOM Data Transport - Linux PC

Issue/Introduction

Linux servers that were previously running Linux 6.9 (Santiago) were upgraded to RHEL 7.7 (Maipo).
After the upgrade testing from the RHEL 7.7 source XCOM server to a RHEL 6.10 target XCOM server with command "xcomtcp -ping" on secure socket (8045) shows SSL handshake error as follows:

RHEL 7.7 source:
# xcomtcp -ping REMOTE_SYSTEM=xxx PORT=8045 TRNENCRL_CIPHER=ALL SECURE_SOCKET=YES
XCOMU0298E Unable to allocate remote transaction program: Txpi 215: Socket send error return value = 9

RHEL 6.10 target:
The $XCOM_HOME/xcom.log file contains this corresponding error:
XCOMU0780E Txpi 308: TxpiInitSSL Failed msg = <error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
+++ SSL alert number 40> value = 0:

Resolution

XCOM support not able to recreate "xcomtcp -ping" problem from RHEL 7.7 to RHEL 6.10 and advised to run the "xcomtcp -ping" command with trace options "-t" and "XTRACE=10" as follows:
# xcomtcp -ping -t REMOTE_SYSTEM=xxx PORT=8045 TRNENCRL_CIPHER=ALL SECURE_SOCKET=YES XTRACE=10 > xcom_ping.out 2>&1
The resulting xcom_ping.out file showed these errors:
===
E - catoossl.c(2899): Error 308

E -catoossl.c(2899): Txpi 308: TxpiInitSSL Failed msg = <error:1408D177:SSL routines:ssl3_get_key_exchange:bad dh g value> value = 4294967295
===

Running openssl command "openssl s_client -connect target_server:8045 -showcerts -status" to the target server also showed error "ssl3_get_key_exchange:bad dh g value" and "Cipher is (NONE)"
XCOM Support observed the same result using openssl command from RHEL 7.7 to RHEL 6.10:
"openssl s_client -connect target_server:8045 -showcerts -status -msg"
"-msg" shows each step of the handshake sequence and the "bad dh g value" error occurs after the server key exchange step:
ClientHello
ServerHello
Certificate
ServerKeyExchange

NOTE: XCOM uses its own CAPKI SSL libraries and not the system openssl libraries.

On the target server, which is effectively the SSL server, the configssl.cnf file [RSAKEY] parameter "RECEIVE_SIDE" controls whether RSA or Diffie Hellman (DH) key exchange is used by XCOM as follows:
1. The OOTB default value for [RSAKEY] parameter "RECEIVE_SIDE" is "NO" which means Diffie Hellman key exchange will be used by the XCOM receiver/target server and the resulting cipher observed by XCOM support in their trace file was DHE-RSA-AES256-SHA. For the reported problems "RECEIVE_SIDE = NO" was being used.
2. When [RSAKEY] parameter "RECEIVE_SIDE" is changed to "YES" it means an RSA key exchange will be used by the XCOM receiver/target server and the resulting cipher observed by XCOM support in their trace file was AES256-SHA. After the option "RECEIVE_SIDE = YES" was implemented it resolved both the "xcomtcp -ping" and openssl errors. Therefore it appears that the RHEL 7.7 upgrade had somehow introduced a problem which prevented use of the Diffie Hellman key exchange that did not exist before. Additional investigation using [RSAKEY] parameter "RECEIVE_SIDE = NO" found that DH could be used successfully for both "xcomtcp -ping" and openssl tests if a prime of 1024 instead of the default 256 was used

So the 2 possible resolutions for target server config.ssl.cnf file are:
1. Set [RSAKEY] parameter RECEIVE_SIDE to YES so that RSA key exchange is used i.e.
===
[RSAKEY]
RECEIVE_SIDE = YES
===

2. Set [RSAKEY] parameter RECEIVE_SIDE to NO so that DH key exchange is used and also use a DH prime of 1024 i.e.
===
[RSAKEY]
RECEIVE_SIDE = NO

# Optional (for RSA NO, see above). If RSA NO and DH files empty,
# then internal program tables will be used.
[DH]
# DH_512_RECEIVE_SIDE = /opt/CA/XCOM/config/dh512.pem
DH_1024_RECEIVE_SIDE = /opt/CA/XCOM/config/dh1024.pem
# DH_2048_RECEIVE_SIDE = /opt/CA/XCOM/config/dh2048.pem
# DH_4096_RECEIVE_SIDE = /opt/CA/XCOM/config/dh4096.pem
===

However, as RSA is more secure than DH it is normally advised to use the RSA option.